diff --git a/docs/README.md b/docs/README.md index d177340078..94cf5d9660 100644 --- a/docs/README.md +++ b/docs/README.md @@ -24,6 +24,7 @@ This section points to sources that explain why ServiceControl is designed the w - [Ingestion pipeline](ingestion-pipeline.md) — why batch parallelism is a storage decision, not an instance decision - [Error ingestion design](error-ingestion-design.md) — why the relational persisters write failed messages with hand-written SQL +- [Message body storage](message-body-storage.md) — where the Audit SQL Server and PostgreSQL persisters keep message bodies, how retention removes them, and which settings are required - [Bulk retries design](bulk-retries-design.md) — how ServiceControl retries failed messages in bulk - [Retries over Azure Storage Queues transport](retries-asq-transport.md) — how retries behave with one or several storage accounts - [Data versioning design](data-versioning-design.md) — which invariant keeps a cached API response from going stale diff --git a/docs/load-testing.md b/docs/load-testing.md index a16ac48585..67fa78a888 100644 --- a/docs/load-testing.md +++ b/docs/load-testing.md @@ -21,6 +21,11 @@ Useful flags, passed after `--`: | `--tag ` | Test a specific ServiceControl image tag, for example a PR prerelease like `pr-1234` | | `--persistence ` | Persistence for the error instance: `RavenDb` (default), `SqlServer`, or `PostgreSql` | | `--error-ingestion-scale-unit ` | Start `n` additional error-ingestion-only ServiceControl instances | +| `--azureblob-body-storage ""` | Store message bodies in Azure Blob Storage instead of the container file system. Applies only to `SqlServer` and `PostgreSql`. The error instance and its scale-out units share one container, and each audit instance gets its own | +| `--s3-body-storage ` | Store message bodies in S3 instead of the container file system. Applies only to `SqlServer` and `PostgreSql`, and is mutually exclusive with `--azureblob-body-storage`. The error instance and its scale-out units share the `error-bodies/` prefix, and each audit instance gets its own | +| `--s3-region ` | Region for `--s3-body-storage` | +| `--s3-service-url ` | S3-compatible endpoint, such as MinIO or LocalStack. It must be reachable from inside the containers, so `localhost` does not work | +| `--s3-access-key-id `, `--s3-secret-access-key ` | Static credentials, set both or neither. Without them, the containers use the default AWS credential chain; host credentials are not forwarded | ## Run against an existing ServiceControl @@ -35,7 +40,7 @@ Open http://localhost:5290. The tool defaults to a ServiceControl instance at `h Everything is controllable from the web UI, or via the HTTP API: -- Scenarios: the five built-in failure scenarios (`third-party-outage`, `timeout-spike`, `poison-message`, `deserialization-failure`, `background-noise`), each producing errors that group naturally. Start with `POST /api/scenarios/{name}/start` (optional `rate` and `durationSeconds`). +- Scenarios: the six built-in scenarios (`third-party-outage`, `timeout-spike`, `poison-message`, `deserialization-failure`, `background-noise`, `large-messages`), each producing errors that group naturally. `large-messages` starts at 3 messages per second unless `rate` is set, sends bodies of about 256 KiB (the wire envelope is slightly larger), fails 20% of them to the error queue, and audits the rest. The bypass writer fails every message by design, so it sends the large bodies only as failures. Start with `POST /api/scenarios/{name}/start` (optional `rate` and `durationSeconds`). - Bypass writer: high-throughput load that writes failed-message envelopes directly to the ServiceControl error queue, skipping the message handler (`POST /api/bypass/start`). - Jobs: retry, archive, search, retention sweep, and custom-check-failure jobs exercise ServiceControl's recoverability, FTS search, and retention pipelines against the load. Jobs do not start automatically. Start them from the UI or `/api/jobs`. - Release-test presets: named presets mapped from the [testing scenarios](testing-scenarios.md), for example `POST /api/release-tests/ingestion-load/start`. diff --git a/docs/message-body-storage.md b/docs/message-body-storage.md new file mode 100644 index 0000000000..6f6d25cfbe --- /dev/null +++ b/docs/message-body-storage.md @@ -0,0 +1,79 @@ +# Message body storage + +The SQL Server and PostgreSQL persisters for ServiceControl.Audit keep every message body. A small valid text body stays in the database row. Any other body goes to external storage: a directory, an Azure Blob container, or an S3 bucket. The providers live in `src/ServiceControl.Persistence.BodyStorage` and are shared with the Error (primary) persisters. + +## Placement rules + +`MaxBodySizeToStore` (default 100 KB) is the inline threshold. It does not limit the size of a stored body. + +| Body | Where it goes | `BodyState` | Inline `BodyText` | +| --- | --- | --- | --- | +| Empty | Nowhere | `None` | none | +| Valid UTF-8 text, at most `MaxBodySizeToStore` bytes | Database row | `Stored` | whole body | +| Valid UTF-8 text, over `MaxBodySizeToStore` bytes | External storage | `External` | first `MaxBodySizeToStore` bytes, as a search prefix | +| Binary content type, `Content-Encoding` header, invalid UTF-8, or a NUL character | External storage | `External` | none | + +External storage always holds the complete body. Only the inline prefix is searchable, as before. + +Rows written before external storage existed keep their meaning: `TooLarge` answers 204 and `NotText` answers 404. The bodies behind those rows were discarded at ingestion and cannot be recovered. `External` is a new value of the existing integer column, so there is no schema change. + +## Ordering and failures + +Ingestion writes external bodies before the batch transaction that inserts the rows. A failed write fails the batch, and no row is inserted. A failed insert can leave an unreferenced body. + +A body is stored under its audit body id, `yyyyMMddHH-{unique message id}`, the same id used in the body URL. Storage keeps the first write of an id and ignores later writes. + +Reading a body streams it from storage with the stored content type, size, and a version derived from the row. A body missing from storage answers 204. + +## Retention + +Retention removes bodies before it removes rows. For each expired unit it calls `IBodyStoragePersistence.DeleteBodiesWithPrefix`, and it drops the rows only after that call succeeds. If storage fails, the rows and partitions stay and the next sweep retries. + +The unit differs by provider, so the prefix is part of `IAuditPartitionManager`: + +- SQL Server deletes rows one hour at a time. The prefix is the hour, `yyyyMMddHH-`. +- PostgreSQL drops whole day partitions without scanning rows. The prefix is the day, `yyyyMMdd`, and it matches every hour of that day. A day is only dropped once all of it is older than the retention period, so no body is removed earlier than its row would be. + +Prefixes must not be empty and must not contain wildcard or path characters. SQL Server only visits hours that have rows, so an unreferenced body in an hour with no rows stays until that location is cleaned manually. PostgreSQL visits every expired day partition, including pre-created partitions without rows, so it also cleans unreferenced bodies of those days. + +## Configuration + +External storage is required. The instance fails to start when `MessageBody/StorageType` is missing, which includes existing SQL Server and PostgreSQL Audit instances that upgrade without adding it. Setup creates the directory, container, or bucket. + +Settings use the `ServiceControl.Audit/` prefix in the application settings file. The validation matches the Error instance, whose settings use the `ServiceControl/` prefix. + +| Key | Applies to | Default | +| --- | --- | --- | +| `MessageBody/StorageType` | all, required: `FileSystem`, `AzureBlob`, or `S3` | none | +| `MessageBody/MinCompressionSize` | all, minimum body size in bytes to compress | 4096 | +| `MessageBody/FileSystem/StoragePath` | `FileSystem`, required | none | +| `MessageBody/FileSystem/DataSpaceRemainingThreshold` | `FileSystem`, free space percentage below which the custom check fails | 15 | +| `MessageBody/Azure/ConnectionString` | `AzureBlob`, shared key or SAS | none | +| `MessageBody/Azure/ServiceUri` | `AzureBlob`, managed identity | none | +| `MessageBody/Azure/ManagedIdentityClientId` | `AzureBlob`, managed identity | none | +| `MessageBody/Azure/AuthorityHost` | `AzureBlob`, managed identity | SDK default | +| `MessageBody/Azure/ContainerName` | `AzureBlob` | `audit-bodies` | +| `MessageBody/S3/BucketName` | `S3`, required | none | +| `MessageBody/S3/KeyPrefix` | `S3` | `audit-bodies/` | +| `MessageBody/S3/Region` | `S3` | SDK default | +| `MessageBody/S3/ServiceUrl` | `S3` | SDK default | +| `MessageBody/S3/AccessKeyId`, `MessageBody/S3/SecretAccessKey` | `S3`, set both or neither | default AWS credential chain | + +Azure requires exactly one of the connection string and the service URI. `MaxBodySizeToStore` keeps its existing top-level setting. + +```xml + + +``` + +```xml + + + +``` + +## One location per instance + +Give each Audit instance its own directory, container, or key prefix, and never share one with an Error instance. Retention deletes by id prefix, and the prefixes of two instances can match the same hours. An instance that shares a location can delete bodies that another instance still references. The Audit defaults differ from the Error defaults (`error-bodies`, `error-bodies/`) for this reason, but they are the same for every Audit instance, so a second Audit instance in the same account must set its own container name or key prefix. + +With `FileSystem` storage, an instance registers the `ServiceControl body storage` custom check that reports low free space on the drive. diff --git a/src/ServiceControl.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs b/src/ServiceControl.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs index 786d9b4cb4..08722fe8d0 100644 --- a/src/ServiceControl.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs +++ b/src/ServiceControl.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs @@ -1,5 +1,6 @@ namespace ServiceControl.AcceptanceTests.PostgreSql; +using ServiceControl.Persistence.BodyStorage; using System; using System.Collections.Concurrent; using System.IO; diff --git a/src/ServiceControl.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs b/src/ServiceControl.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs index fd453c930a..1f8fe026e3 100644 --- a/src/ServiceControl.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs +++ b/src/ServiceControl.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs @@ -1,5 +1,6 @@ namespace ServiceControl.AcceptanceTests.SqlServer; +using ServiceControl.Persistence.BodyStorage; using System; using System.Collections.Concurrent; using System.IO; diff --git a/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs b/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs index 09977fdebe..7dba9a1c84 100644 --- a/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs +++ b/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs @@ -3,10 +3,11 @@ namespace ServiceControl.AcceptanceTests using System; using System.Runtime.Loader; using System.Threading.Tasks; + using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.DependencyInjection; - using Microsoft.Extensions.Hosting; using NServiceBus; using NUnit.Framework; + using global::ServiceControl.Infrastructure.WebApi; using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; @@ -41,9 +42,10 @@ public void Should_keep_the_clock_the_host_registered() var endpointConfiguration = new EndpointConfiguration(settings.InstanceName); endpointConfiguration.AssemblyScanner().Disable = true; - var hostBuilder = Host.CreateApplicationBuilder(); + var hostBuilder = WebApplication.CreateBuilder(); hostBuilder.Services.AddSingleton(hostClock); hostBuilder.AddServiceControl(settings, endpointConfiguration); + hostBuilder.AddServiceControlApi(settings.CorsSettings); using var host = hostBuilder.Build(); diff --git a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs index c6aeb6f385..82c9ed933b 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs +++ b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Audit.AcceptanceTests.PostgreSql; using System; using System.Collections.Concurrent; using System.Collections.Generic; +using System.IO; using System.Threading; using System.Threading.Tasks; using ServiceControl.Audit.AcceptanceTests.TestSupport; @@ -22,18 +23,39 @@ public async Task> CustomizeSettings(CancellationTok schemas.Add(schema); + var bodyStoragePath = Directory.CreateTempSubdirectory("sc_audit_at_bodies_").FullName; + bodyStoragePaths.Add(bodyStoragePath); + return new Dictionary { [EFPersistenceConfigurationBase.ConnectionStringKey] = connectionString, - [EFPersistenceConfigurationBase.SchemaKey] = schema + [EFPersistenceConfigurationBase.SchemaKey] = schema, + ["MessageBody/StorageType"] = "FileSystem", + ["MessageBody/FileSystem/StoragePath"] = bodyStoragePath }; } public async Task Cleanup(CancellationToken cancellationToken = default) { - while (schemas.TryTake(out var schema)) + try + { + while (schemas.TryTake(out var schema)) + { + await TestSchema.Drop(connectionString, schema, cancellationToken); + } + } + finally { - await TestSchema.Drop(connectionString, schema, cancellationToken); + while (bodyStoragePaths.TryTake(out var bodyStoragePath)) + { + try + { + Directory.Delete(bodyStoragePath, recursive: true); + } + catch (DirectoryNotFoundException) + { + } + } } } @@ -51,5 +73,6 @@ sealed class Release : IDisposable static readonly SemaphoreSlim initializationLock = new(1, 1); readonly ConcurrentBag schemas = []; + readonly ConcurrentBag bodyStoragePaths = []; string connectionString; } diff --git a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj index 4836ca191d..01af31ab2c 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj +++ b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj @@ -30,12 +30,16 @@ + + + + diff --git a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs index 52485fc4a1..54c4b0a87d 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs +++ b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Audit.AcceptanceTests.SqlServer; using System; using System.Collections.Concurrent; using System.Collections.Generic; +using System.IO; using System.Threading; using System.Threading.Tasks; using ServiceControl.Audit.AcceptanceTests.TestSupport; @@ -22,18 +23,39 @@ public async Task> CustomizeSettings(CancellationTok schemas.Add(schema); + var bodyStoragePath = Directory.CreateTempSubdirectory("sc_audit_at_bodies_").FullName; + bodyStoragePaths.Add(bodyStoragePath); + return new Dictionary { [EFPersistenceConfigurationBase.ConnectionStringKey] = connectionString, - [EFPersistenceConfigurationBase.SchemaKey] = schema + [EFPersistenceConfigurationBase.SchemaKey] = schema, + ["MessageBody/StorageType"] = "FileSystem", + ["MessageBody/FileSystem/StoragePath"] = bodyStoragePath }; } public async Task Cleanup(CancellationToken cancellationToken = default) { - while (schemas.TryTake(out var schema)) + try + { + while (schemas.TryTake(out var schema)) + { + await TestSchema.Drop(connectionString, schema, cancellationToken); + } + } + finally { - await TestSchema.Drop(connectionString, schema, cancellationToken); + while (bodyStoragePaths.TryTake(out var bodyStoragePath)) + { + try + { + Directory.Delete(bodyStoragePath, recursive: true); + } + catch (DirectoryNotFoundException) + { + } + } } } @@ -51,5 +73,6 @@ sealed class Release : IDisposable static readonly SemaphoreSlim initializationLock = new(1, 1); readonly ConcurrentBag schemas = []; + readonly ConcurrentBag bodyStoragePaths = []; string connectionString; } diff --git a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj index 0294b97928..0ed800edbf 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj +++ b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj @@ -36,6 +36,9 @@ + + + diff --git a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/When_messages_with_big_bodies_are_ingested.ExternalBody.cs b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/When_messages_with_big_bodies_are_ingested.ExternalBody.cs new file mode 100644 index 0000000000..3d4f3f91e8 --- /dev/null +++ b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/When_messages_with_big_bodies_are_ingested.ExternalBody.cs @@ -0,0 +1,49 @@ +namespace ServiceControl.Audit.AcceptanceTests.Auditing +{ + using System.Text.Json; + using System.Threading.Tasks; + using AcceptanceTesting; + using AcceptanceTesting.EndpointTemplates; + using Audit.Auditing.MessagesView; + using NServiceBus; + using NServiceBus.AcceptanceTesting; + using NUnit.Framework; + + partial class When_messages_with_big_bodies_are_ingested + { + [Test] + public async Task Should_serve_the_full_body_when_it_is_larger_than_MaxBodySizeToStore() + { + SetSettings = settings => settings.MaxBodySizeToStore = MAX_BODY_SIZE; + + var expected = new byte[MAX_BODY_SIZE + 1000]; + byte[] body = null; + + await Define() + .WithEndpoint(c => c.When(b => b.SendLocal(new BigFatMessage { BigFatBody = expected }))) + .Done( + async c => + { + if (c.MessageId == null) + { + return false; + } + + var result = await this.TryGetSingle("/api/messages", r => r.MessageId == c.MessageId); + MessagesView auditMessage = result; + if (!result) + { + return false; + } + + body = await this.DownloadData($"/api{auditMessage.BodyUrl}"); + + return true; + }) + .Run(); + + using var document = JsonDocument.Parse(body); + Assert.That(document.RootElement.GetProperty(nameof(BigFatMessage.BigFatBody)).GetBytesFromBase64(), Is.EqualTo(expected)); + } + } +} diff --git a/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.MaxBodySize.cs b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.MaxBodySize.cs new file mode 100644 index 0000000000..a01155630b --- /dev/null +++ b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.MaxBodySize.cs @@ -0,0 +1,55 @@ +namespace ServiceControl.Audit.AcceptanceTests.Auditing +{ + using System.Net; + using System.Threading.Tasks; + using AcceptanceTesting; + using AcceptanceTesting.EndpointTemplates; + using Audit.Auditing.MessagesView; + using NServiceBus; + using NServiceBus.AcceptanceTesting; + using NUnit.Framework; + + partial class When_messages_with_big_bodies_are_ingested + { + [Test] + public async Task Should_get_an_empty_audit_message_body_when_configured_MaxBodySizeToStore_is_less_then_message_size() + { + //Arrange + SetSettings = settings => settings.MaxBodySizeToStore = MAX_BODY_SIZE; + + byte[] body = null; + + //Act + await Define() + .WithEndpoint(c => c.When(b => b.SendLocal( + new BigFatMessage // An endpoint that is configured for audit + { + BigFatBody = new byte[MAX_BODY_SIZE + 1000] + })) + ) + .Done( + async c => + { + if (c.MessageId == null) + { + return false; + } + + var result = await this.TryGetSingle("/api/messages", r => r.MessageId == c.MessageId); + MessagesView auditMessage = result; + if (!result) + { + return false; + } + + body = await this.DownloadData($"/api{auditMessage.BodyUrl}", HttpStatusCode.NoContent); + + return true; + }) + .Run(); + + //Assert + Assert.That(body, Is.Empty); + } + } +} diff --git a/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs index 990b705566..5c4d7fa55b 100644 --- a/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs +++ b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs @@ -9,7 +9,7 @@ using NServiceBus.AcceptanceTesting; using NUnit.Framework; - class When_messages_with_big_bodies_are_ingested : AcceptanceTest + partial class When_messages_with_big_bodies_are_ingested : AcceptanceTest { [Test] public async Task Should_not_get_an_empty_audit_message_body_when_configured_MaxBodySizeToStore_is_greater_then_message_size() @@ -52,47 +52,6 @@ await Define() Assert.That(body, Is.Not.Null); } - [Test] - public async Task Should_get_an_empty_audit_message_body_when_configured_MaxBodySizeToStore_is_less_then_message_size() - { - //Arrange - SetSettings = settings => settings.MaxBodySizeToStore = MAX_BODY_SIZE; - - byte[] body = null; - - //Act - await Define() - .WithEndpoint(c => c.When(b => b.SendLocal( - new BigFatMessage // An endpoint that is configured for audit - { - BigFatBody = new byte[MAX_BODY_SIZE + 1000] - })) - ) - .Done( - async c => - { - if (c.MessageId == null) - { - return false; - } - - var result = await this.TryGetSingle("/api/messages", r => r.MessageId == c.MessageId); - MessagesView auditMessage = result; - if (!result) - { - return false; - } - - body = await this.DownloadData($"/api{auditMessage.BodyUrl}", HttpStatusCode.NoContent); - - return true; - }) - .Run(); - - //Assert - Assert.That(body, Is.Empty); - } - [Test] public async Task Should_not_get_an_empty_audit_message_body_when_body_is_above_loh_but_below_max_body_size() { diff --git a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs index 1c310dd52f..77a42d97bf 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs @@ -52,6 +52,9 @@ public async Task> ListExpired(AuditDbContext dbContext, return [.. days]; } + // A partition holds a whole day, so the prefix is the day rather than the hour. + public string BodyIdPrefix(DateTime day) => AuditBodyId.DayPrefix(day); + public async Task DropExpired(AuditDbContext dbContext, DateTime day, int batchSize, CancellationToken cancellationToken = default) { var sql = string.Concat(Tables(dbContext).Select(table => $"DROP TABLE IF EXISTS {table.Partition(day)};\n")); diff --git a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/ServiceControl.Audit.Persistence.EFCore.PostgreSql.csproj b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/ServiceControl.Audit.Persistence.EFCore.PostgreSql.csproj index fce1f0a48a..c8514bbb72 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/ServiceControl.Audit.Persistence.EFCore.PostgreSql.csproj +++ b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/ServiceControl.Audit.Persistence.EFCore.PostgreSql.csproj @@ -26,6 +26,7 @@ + diff --git a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest index abff0adcaf..d2a6287d02 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest +++ b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest @@ -16,6 +16,66 @@ { "Name": "ServiceControl.Audit/Database/CommandTimeout", "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/StorageType", + "Mandatory": true + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/StoragePath", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/DataSpaceRemainingThreshold", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ConnectionString", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ServiceUri", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ManagedIdentityClientId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/AuthorityHost", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ContainerName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/BucketName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/KeyPrefix", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/Region", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/ServiceUrl", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/AccessKeyId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/SecretAccessKey", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/MinCompressionSize", + "Mandatory": false } ] } diff --git a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/ServiceControl.Audit.Persistence.EFCore.SqlServer.csproj b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/ServiceControl.Audit.Persistence.EFCore.SqlServer.csproj index 19212afd31..eba1474634 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/ServiceControl.Audit.Persistence.EFCore.SqlServer.csproj +++ b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/ServiceControl.Audit.Persistence.EFCore.SqlServer.csproj @@ -26,6 +26,7 @@ + diff --git a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs index 65e436b7f1..592ba16e12 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs @@ -18,6 +18,8 @@ public async Task> ListExpired(AuditDbContext dbContext, return await messageHours.Union(snapshotHours).OrderBy(hour => hour).ToListAsync(cancellationToken); } + public string BodyIdPrefix(DateTime hour) => AuditBodyId.HourPrefix(hour); + public async Task DropExpired(AuditDbContext dbContext, DateTime hour, int batchSize, CancellationToken cancellationToken = default) { var messagesDeleted = await dbContext.AuditMessages diff --git a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest index 8c7628c10f..87ae4d11d5 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest +++ b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest @@ -16,6 +16,66 @@ { "Name": "ServiceControl.Audit/Database/CommandTimeout", "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/StorageType", + "Mandatory": true + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/StoragePath", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/DataSpaceRemainingThreshold", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ConnectionString", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ServiceUri", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ManagedIdentityClientId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/AuthorityHost", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ContainerName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/BucketName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/KeyPrefix", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/Region", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/ServiceUrl", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/AccessKeyId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/SecretAccessKey", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/MinCompressionSize", + "Mandatory": false } ] } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettingsParser.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettingsParser.cs new file mode 100644 index 0000000000..b1c2f08f2f --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettingsParser.cs @@ -0,0 +1,174 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; + +using Microsoft.Extensions.Logging; +using ServiceControl.Infrastructure; +using ServiceControl.Persistence.BodyStorage; + +// Mirrors the Error persister's MessageBody/* parsing, which reads a settings root rather than a dictionary. +static class BodyStorageSettingsParser +{ + internal const string StorageTypeKey = "MessageBody/StorageType"; + internal const string FileSystemStoragePathKey = "MessageBody/FileSystem/StoragePath"; + internal const string FileSystemDataSpaceRemainingThresholdKey = "MessageBody/FileSystem/DataSpaceRemainingThreshold"; + internal const string AzureConnectionStringKey = "MessageBody/Azure/ConnectionString"; + internal const string AzureServiceUriKey = "MessageBody/Azure/ServiceUri"; + internal const string AzureManagedIdentityClientIdKey = "MessageBody/Azure/ManagedIdentityClientId"; + internal const string AzureAuthorityHostKey = "MessageBody/Azure/AuthorityHost"; + internal const string AzureContainerNameKey = "MessageBody/Azure/ContainerName"; + internal const string S3BucketNameKey = "MessageBody/S3/BucketName"; + internal const string S3KeyPrefixKey = "MessageBody/S3/KeyPrefix"; + internal const string S3RegionKey = "MessageBody/S3/Region"; + internal const string S3ServiceUrlKey = "MessageBody/S3/ServiceUrl"; + internal const string S3AccessKeyIdKey = "MessageBody/S3/AccessKeyId"; + internal const string S3SecretAccessKeyKey = "MessageBody/S3/SecretAccessKey"; + internal const string MinCompressionSizeKey = "MessageBody/MinCompressionSize"; + + // Audit instances must not share a location with Error or with each other, so the Audit defaults differ from the shared ones. + internal const string DefaultAzureContainerName = "audit-bodies"; + internal const string DefaultS3KeyPrefix = "audit-bodies/"; + + public static readonly string[] Keys = + [ + StorageTypeKey, + FileSystemStoragePathKey, + FileSystemDataSpaceRemainingThresholdKey, + AzureConnectionStringKey, + AzureServiceUriKey, + AzureManagedIdentityClientIdKey, + AzureAuthorityHostKey, + AzureContainerNameKey, + S3BucketNameKey, + S3KeyPrefixKey, + S3RegionKey, + S3ServiceUrlKey, + S3AccessKeyIdKey, + S3SecretAccessKeyKey, + MinCompressionSizeKey + ]; + + public static BodyStorageSettings Parse(IDictionary specific, int maxBodySizeToStore) + { + BodyStorageSettings bodyStorage = ReadStorageType(specific) switch + { + BodyStorageType.FileSystem => new FileSystemBodyStorageSettings + { + StoragePath = GetRequired(specific, FileSystemStoragePathKey), + DataSpaceRemainingThreshold = ReadDataSpaceRemainingThreshold(specific) + }, + BodyStorageType.AzureBlob => new AzureBlobBodyStorageSettings + { + Authentication = ReadAzureBlobAuthentication(specific), + ContainerName = Get(specific, AzureContainerNameKey) ?? DefaultAzureContainerName + }, + BodyStorageType.S3 => new S3BodyStorageSettings + { + BucketName = GetRequired(specific, S3BucketNameKey), + KeyPrefix = Get(specific, S3KeyPrefixKey) ?? DefaultS3KeyPrefix, + Region = Get(specific, S3RegionKey), + ServiceUrl = Get(specific, S3ServiceUrlKey), + Credentials = ReadS3Credentials(specific) + }, + var unknown => throw new ArgumentOutOfRangeException(nameof(specific), unknown, "Unknown body storage type.") + }; + + bodyStorage.MinCompressionSize = ReadInt(specific, MinCompressionSizeKey, BodyStorageSettings.DefaultMinCompressionSize); + bodyStorage.MaxBodySizeToStore = ReadMaxBodySizeToStore(maxBodySizeToStore); + + return bodyStorage; + } + + static BodyStorageType ReadStorageType(IDictionary specific) + { + var raw = Get(specific, StorageTypeKey); + + if (raw is null) + { + throw new InvalidOperationException($"{StorageTypeKey} is required. Valid values: {string.Join(", ", Enum.GetNames())}."); + } + + if (!Enum.TryParse(raw, ignoreCase: true, out var value) || !Enum.IsDefined(value)) + { + throw new InvalidOperationException($"{StorageTypeKey} value '{raw}' is not valid. Valid values: {string.Join(", ", Enum.GetNames())}."); + } + + return value; + } + + static AzureBlobAuthentication ReadAzureBlobAuthentication(IDictionary specific) + { + var connectionString = Get(specific, AzureConnectionStringKey); + var serviceUri = Get(specific, AzureServiceUriKey); + + // A connection string carries shared-key/SAS auth; a service URI uses managed identity. + if ((connectionString is null) == (serviceUri is null)) + { + throw new InvalidOperationException($"Azure Blob body storage requires exactly one of {AzureConnectionStringKey} (shared key / SAS) or {AzureServiceUriKey} (managed identity). {(connectionString is null ? "Neither was set." : "Both were set.")}"); + } + + if (connectionString is not null) + { + return new AzureBlobSharedKeyAuthentication { ConnectionString = connectionString }; + } + + var authorityHost = Get(specific, AzureAuthorityHostKey); + + return new AzureBlobManagedIdentityAuthentication + { + ServiceUri = ReadAbsoluteUri(serviceUri!, AzureServiceUriKey), + ClientId = Get(specific, AzureManagedIdentityClientIdKey), + AuthorityHost = authorityHost is null ? null : ReadAbsoluteUri(authorityHost, AzureAuthorityHostKey) + }; + } + + static S3StaticCredentials? ReadS3Credentials(IDictionary specific) + { + var accessKeyId = Get(specific, S3AccessKeyIdKey); + var secretAccessKey = Get(specific, S3SecretAccessKeyKey); + + if ((accessKeyId is null) != (secretAccessKey is null)) + { + throw new InvalidOperationException($"S3 body storage requires both {S3AccessKeyIdKey} and {S3SecretAccessKeyKey} together, or neither to use the default AWS credential chain (IAM role)."); + } + + return accessKeyId is null ? null : new S3StaticCredentials { AccessKeyId = accessKeyId, SecretAccessKey = secretAccessKey! }; + } + + static Uri ReadAbsoluteUri(string raw, string key) => + Uri.TryCreate(raw, UriKind.Absolute, out var uri) + ? uri + : throw new InvalidOperationException($"{key} value '{raw}' is not a valid absolute URI."); + + static int ReadDataSpaceRemainingThreshold(IDictionary specific) + { + var threshold = ReadInt(specific, FileSystemDataSpaceRemainingThresholdKey, FileSystemBodyStorageSettings.DefaultDataSpaceRemainingThreshold); + + return threshold is < 0 or > 100 + ? throw new InvalidOperationException($"{FileSystemDataSpaceRemainingThresholdKey} value '{threshold}' is not valid. The value is a percentage between 0 and 100.") + : threshold; + } + + static int ReadMaxBodySizeToStore(int maxBodySizeToStore) + { + if (maxBodySizeToStore > 0) + { + return maxBodySizeToStore; + } + + logger.LogError("MaxBodySizeToStore setting is invalid, 1 is the minimum value. Defaulting to {MaxBodySizeToStoreDefault}", BodyStorageSettings.DefaultMaxBodySizeToStore); + + return BodyStorageSettings.DefaultMaxBodySizeToStore; + } + + static int ReadInt(IDictionary specific, string key, int defaultValue) => + Get(specific, key) is not { } value ? defaultValue + : int.TryParse(value, out var parsed) ? parsed + : throw new InvalidOperationException($"{key} must be an integer."); + + static string GetRequired(IDictionary specific, string key) => + Get(specific, key) ?? throw new InvalidOperationException($"{key} must be specified."); + + static string? Get(IDictionary specific, string key) => + specific.TryGetValue(key, out var value) && !string.IsNullOrWhiteSpace(value) ? value : null; + + static readonly ILogger logger = LoggerUtil.CreateStaticLogger(); +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs index 90cb94678b..f11c4d2b14 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; using NServiceBus; +using ServiceControl.Persistence.BodyStorage; using ServiceControl.Audit.Persistence.EFCore.Implementation; using ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Audit.Persistence.UnitOfWork; @@ -13,6 +14,7 @@ public void AddPersistence(IServiceCollection services) { AddCommon(services); AddQueryServices(services); + AddBodyStorage(services); services.AddSingleton(); services.AddSingleton(provider => provider.GetRequiredService()); @@ -32,6 +34,11 @@ public void AddPersistence(IServiceCollection services) { endpointConfiguration.AddCustomCheck(); AddCustomChecks(endpointConfiguration); + + if (settings.BodyStorage is FileSystemBodyStorageSettings) + { + endpointConfiguration.AddCustomCheck(); + } } } @@ -49,6 +56,29 @@ void AddCommon(IServiceCollection services) AddPartitionManager(services); } + // The settings type picks the store, so each store resolves only the settings it can act on. + void AddBodyStorage(IServiceCollection services) + { + switch (settings.BodyStorage) + { + case FileSystemBodyStorageSettings fileSystem: + services.AddSingleton(fileSystem); + services.AddSingleton(); + services.AddSingleton(); + break; + case AzureBlobBodyStorageSettings azureBlob: + services.AddSingleton(azureBlob); + services.AddSingleton(); + break; + case S3BodyStorageSettings s3: + services.AddSingleton(s3); + services.AddSingleton(); + break; + default: + throw new ArgumentOutOfRangeException(nameof(settings), settings.BodyStorage, "Unknown body storage type."); + } + } + protected abstract void AddDbContext(IServiceCollection services, EFPersisterSettings settings); protected abstract void AddPartitionManager(IServiceCollection services); diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs index 4b1bb5da46..4bc5453a81 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs @@ -15,7 +15,7 @@ public abstract class EFPersistenceConfigurationBase : IPersistenceConfiguration public abstract string Name { get; } - public IEnumerable ConfigurationKeys => [ConnectionStringKey, SchemaKey, CommandTimeoutKey, QueryTimeoutInSecondsKey]; + public IEnumerable ConfigurationKeys => [ConnectionStringKey, SchemaKey, CommandTimeoutKey, QueryTimeoutInSecondsKey, .. BodyStorageSettingsParser.Keys]; public IPersistence Create(PersistenceSettings settings) => Create(CreateSettings(settings)); @@ -37,7 +37,7 @@ public static EFPersisterSettings CreateSettings(PersistenceSettings settings) CommandTimeout = ReadInt(specific, CommandTimeoutKey, EFPersisterSettings.DefaultCommandTimeout), QueryTimeout = QueryTimeLimit.Validate(ReadInt(specific, QueryTimeoutInSecondsKey, QueryTimeLimit.DefaultSeconds), QueryTimeoutSettingName, logger), AuditRetentionPeriod = settings.AuditRetentionPeriod, - MaxBodySizeToStore = settings.MaxBodySizeToStore + BodyStorage = BodyStorageSettingsParser.Parse(specific, settings.MaxBodySizeToStore) }; } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs index f441412deb..98b321c0f1 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Infrastructure; +using ServiceControl.Persistence.BodyStorage; public class EFPersisterSettings { @@ -19,7 +20,7 @@ public string? Schema public int CommandTimeout { get; init; } = DefaultCommandTimeout; public TimeSpan QueryTimeout { get; init; } = QueryTimeLimit.Default; public required TimeSpan AuditRetentionPeriod { get; init; } - public required int MaxBodySizeToStore { get; init; } + public required BodyStorageSettings BodyStorage { get; init; } public int MaxRetryCount { get; init; } = 5; public int MaxRetryDelayInSeconds { get; init; } = 30; public bool EnableRetryOnFailure { get; init; } = true; diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs b/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs index 59d773602b..d7d8055d86 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs @@ -5,5 +5,6 @@ public enum BodyState None = 0, Stored = 1, TooLarge = 2, - NotText = 3 + NotText = 3, + External = 4 } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs index 4bd9e8d840..28cfc8e44b 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs @@ -9,12 +9,14 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation; using ServiceControl.Audit.Persistence.EFCore.Entities; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; using ServiceControl.Infrastructure; +using ServiceControl.Persistence.BodyStorage; using ServiceControl.SagaAudit; sealed class AuditDataStore( IServiceScopeFactory scopeFactory, EFPersisterSettings settings, IFullTextSearchDialect fullTextSearch, + IBodyStoragePersistence bodyStorage, TimeProvider timeProvider) : DataStoreBase(scopeFactory, settings), IAuditMessagesViewDataStore, ISagaHistoryDataStore { public const int TotalCountCap = 100_000; @@ -43,8 +45,9 @@ public Task>> QueryMessagesByReceivingEndpoint(b public Task>> QueryMessagesByConversationId(string conversationId, PagingInfo pagingInfo, SortInfo sortInfo, CancellationToken cancellationToken = default) => Page(messages => messages.Where(message => message.ConversationId == conversationId), pagingInfo, sortInfo, cancellationToken); - public Task GetMessageBody(string messageId, CancellationToken cancellationToken = default) => - ExecuteQueryWithDbContext(async (dbContext, token) => + public async Task GetMessageBody(string messageId, CancellationToken cancellationToken = default) + { + var body = await ExecuteQueryWithDbContext(async (dbContext, token) => { IQueryable candidates; if (AuditBodyId.TryParse(messageId, out var createdOn, out var uniqueMessageId)) @@ -57,29 +60,43 @@ public Task GetMessageBody(string messageId, CancellationToken } else { - return MessageBodyView.NotFound(); + return null; } - var body = await candidates + return await candidates .AsNoTracking() .OrderByDescending(message => message.CreatedOn) .ThenByDescending(message => message.Id) - .Select(message => new { message.CreatedOn, message.Id, message.BodyState, message.BodyText, message.BodyContentType, message.BodySize }) + .Select(message => new BodyRow(message.CreatedOn, message.Id, message.UniqueMessageId, message.BodyState, message.BodyText, message.BodyContentType, message.BodySize)) .FirstOrDefaultAsync(token); + }, cancellationToken); - if (body is null) - { - return MessageBodyView.NotFound(); - } + if (body is null) + { + return MessageBodyView.NotFound(); + } - return body.BodyState switch - { - BodyState.Stored => MessageBodyView.FromString(body.BodyText!, body.BodyContentType ?? "text/plain", body.BodySize, DataVersion.Compose(("row", $"{body.CreatedOn.Ticks}-{body.Id}"))), - BodyState.None or BodyState.TooLarge => MessageBodyView.NoContent(), - BodyState.NotText => MessageBodyView.NotFound(), - _ => throw new InvalidOperationException($"Unknown body state {body.BodyState}") - }; - }, cancellationToken); + var version = DataVersion.Compose(("row", $"{body.CreatedOn.Ticks}-{body.Id}")); + + switch (body.BodyState) + { + case BodyState.Stored: + return MessageBodyView.FromString(body.BodyText!, body.BodyContentType ?? "text/plain", body.BodySize, version); + case BodyState.None or BodyState.TooLarge: + return MessageBodyView.NoContent(); + case BodyState.NotText: + return MessageBodyView.NotFound(); + case BodyState.External: + // Read outside the query so the stream is not tied to the query timeout. + var external = await bodyStorage.ReadBody(AuditBodyId.Format(body.CreatedOn, body.UniqueMessageId), cancellationToken); + + return external is null + ? MessageBodyView.NoContent() + : MessageBodyView.FromStream(external.Stream, external.ContentType, external.BodySize, version); + default: + throw new InvalidOperationException($"Unknown body state {body.BodyState}"); + } + } public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) => ExecuteQueryWithDbContext(async (dbContext, token) => @@ -182,4 +199,6 @@ IQueryable Search(IQueryable source, str OutgoingMessages = SagaSnapshotJson.ReadOutgoingMessages(snapshot.OutgoingMessagesJson), Endpoint = snapshot.Endpoint }; + + sealed record BodyRow(DateTime CreatedOn, long Id, Guid UniqueMessageId, BodyState BodyState, string? BodyText, string? BodyContentType, int BodySize); } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs index 588631ee32..90db027ce6 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs @@ -6,10 +6,12 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation; using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.DbContexts; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.BodyStorage; sealed class AuditRetention( IServiceScopeFactory scopeFactory, IAuditPartitionManager partitions, + IBodyStoragePersistence bodyStorage, IRetentionLock retentionLock, AuditRetentionCustomCheck.State retentionState, EFPersisterSettings settings, @@ -114,6 +116,9 @@ async Task Sweep(bool pace, CancellationToken cancellationToken) foreach (var expired in await partitions.ListExpired(dbContext, keepFrom, cancellationToken)) { + // Bodies go first: a failure here leaves the rows in place, so the next sweep retries the same hour. + await bodyStorage.DeleteBodiesWithPrefix(partitions.BodyIdPrefix(expired), cancellationToken); + DropResult drop; do { diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs index e36630c426..1e8597d340 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs @@ -7,15 +7,19 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation; using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.DbContexts; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.BodyStorage; sealed class DatabaseSetup( IServiceScopeFactory scopeFactory, IAuditPartitionManager partitions, + EFPersisterSettings settings, TimeProvider timeProvider, ILogger logger) : IHostedService { public async Task StartAsync(CancellationToken cancellationToken = default) { + await BodyStorageProvisioner.Provision(settings.BodyStorage, cancellationToken); + await using var scope = scopeFactory.CreateAsyncScope(); var dbContext = scope.ServiceProvider.GetRequiredService(); dbContext.Database.SetCommandTimeout(EFPersisterSettings.MigrationCommandTimeout); diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs index ac5b2b928e..2b636049d0 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs @@ -11,20 +11,28 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; using ServiceControl.Audit.Persistence.Infrastructure; using ServiceControl.Audit.Persistence.UnitOfWork; +using ServiceControl.Persistence.BodyStorage; using ServiceControl.SagaAudit; -sealed class AuditIngestionUnitOfWork(IServiceScopeFactory scopeFactory, EFPersisterSettings settings, DateTime createdOn) : IAuditIngestionUnitOfWork +sealed class AuditIngestionUnitOfWork(IServiceScopeFactory scopeFactory, IBodyStoragePersistence bodyStorage, EFPersisterSettings settings, DateTime createdOn) : IAuditIngestionUnitOfWork { readonly List messages = []; readonly List snapshots = []; + readonly List bodyWrites = []; public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default) { var headers = processedMessage.Headers; var metadata = processedMessage.MessageMetadata; var uniqueMessageId = ToGuid(processedMessage.UniqueMessageId); - var (bodyState, bodyText) = MessageBodyClassifier.Classify(headers, body, settings.MaxBodySizeToStore); + var (bodyState, bodyText) = MessageBodyClassifier.Classify(headers, body, settings.BodyStorage.MaxBodySizeToStore); var bodyId = AuditBodyId.Format(createdOn, uniqueMessageId); + var contentType = headers.GetValueOrDefault(Headers.ContentType, "text/plain"); + + if (bodyState == BodyState.External) + { + bodyWrites.Add(bodyStorage.WriteBody(bodyId, body, contentType, cancellationToken)); + } processedMessage.Id ??= bodyId; metadata["ContentLength"] = body.Length; @@ -60,7 +68,7 @@ public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMe BodyText = bodyText, BodyState = bodyState, BodySize = body.Length, - BodyContentType = headers.GetValueOrDefault(Headers.ContentType, "text/plain") + BodyContentType = contentType }); return Task.CompletedTask; @@ -93,6 +101,9 @@ public async Task Complete(CancellationToken cancellationToken = default) return; } + // External bodies are written before the rows that point at them + await Task.WhenAll(bodyWrites); + await using var scope = scopeFactory.CreateAsyncScope(); var dbContext = scope.ServiceProvider.GetRequiredService(); @@ -105,7 +116,7 @@ await dbContext.Database.CreateExecutionStrategy().ExecuteAsync(async token => }, cancellationToken); } - public ValueTask DisposeAsync() => ValueTask.CompletedTask; + public async ValueTask DisposeAsync() => await Task.WhenAll(bodyWrites).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing); static Guid ToGuid(string? uniqueMessageId) => uniqueMessageId is null ? Guid.NewGuid() diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs index 4bae7be84d..5ba817ecc9 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs @@ -4,13 +4,14 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; using ServiceControl.Audit.Persistence.UnitOfWork; +using ServiceControl.Persistence.BodyStorage; -sealed class AuditIngestionUnitOfWorkFactory(IServiceScopeFactory scopeFactory, EFPersisterSettings settings, TimeProvider timeProvider) : IAuditIngestionUnitOfWorkFactory +sealed class AuditIngestionUnitOfWorkFactory(IServiceScopeFactory scopeFactory, IBodyStoragePersistence bodyStorage, EFPersisterSettings settings, TimeProvider timeProvider) : IAuditIngestionUnitOfWorkFactory { // Fixed for the whole batch, so the body ids it records match the rows it writes. public ValueTask StartNew(int batchSize, CancellationToken cancellationToken = default) => ValueTask.FromResult( - new AuditIngestionUnitOfWork(scopeFactory, settings, AuditHours.Truncate(timeProvider.GetUtcNow().UtcDateTime))); + new AuditIngestionUnitOfWork(scopeFactory, bodyStorage, settings, AuditHours.Truncate(timeProvider.GetUtcNow().UtcDateTime))); public bool CanIngestMore() => true; diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs index 6a20dab471..cc5fa798c4 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs @@ -4,6 +4,14 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using NServiceBus; using ServiceControl.Audit.Persistence.EFCore.Entities; +/// +/// Decides whether a body lives inline in the database or in external storage. +/// +/// +/// Bodies are always kept. MaxBodySizeToStore only decides where: inline in BodyText, or in +/// external storage with at most a search prefix left inline. TooLarge and NotText remain for +/// rows written before external storage existed. +/// static class MessageBodyClassifier { public static (BodyState State, string? Text) Classify(IReadOnlyDictionary headers, ReadOnlyMemory body, int maxBodySizeToStore) @@ -14,11 +22,10 @@ public static (BodyState State, string? Text) Classify(IReadOnlyDictionary maxBodySizeToStore; - var notStored = tooLarge ? BodyState.TooLarge : BodyState.NotText; if (!MightBeText(headers)) { - return (notStored, null); + return (BodyState.External, null); } var span = body.Span; @@ -31,16 +38,16 @@ public static (BodyState State, string? Text) Classify(IReadOnlyDictionary headers) diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/AuditBodyId.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/AuditBodyId.cs index d5483d7a64..379bd8c96b 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/AuditBodyId.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/AuditBodyId.cs @@ -5,10 +5,16 @@ namespace ServiceControl.Audit.Persistence.EFCore.Infrastructure; static class AuditBodyId { const string HourFormat = "yyyyMMddHH"; + const string DayFormat = "yyyyMMdd"; public static string Format(DateTime createdOn, Guid uniqueMessageId) => $"{createdOn.ToString(HourFormat, CultureInfo.InvariantCulture)}-{uniqueMessageId}"; + // Ids start with the hour, so a day's bodies share one prefix and a partition drop can match them. + public static string HourPrefix(DateTime hour) => $"{hour.ToString(HourFormat, CultureInfo.InvariantCulture)}-"; + + public static string DayPrefix(DateTime day) => day.ToString(DayFormat, CultureInfo.InvariantCulture); + public static bool TryParse(string bodyId, out DateTime createdOn, out Guid uniqueMessageId) { createdOn = default; diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs index 2b6e7de104..54ac1cb759 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs @@ -8,6 +8,11 @@ interface IAuditPartitionManager Task> ListExpired(AuditDbContext dbContext, DateTime keepFrom, CancellationToken cancellationToken = default); + /// + /// The prefix of every body id stored for the rows that removes for . + /// + string BodyIdPrefix(DateTime expired); + Task DropExpired(AuditDbContext dbContext, DateTime start, int batchSize, CancellationToken cancellationToken = default); Task ProvisionedUntil(AuditDbContext dbContext, CancellationToken cancellationToken = default); diff --git a/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj b/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj index 6b51feb1d8..87c16effae 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj +++ b/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj @@ -15,6 +15,11 @@ + + + + + diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt index 648f28180a..d536054718 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt @@ -1,2 +1,3 @@ ServiceControl.Audit Health: Audit partition provisioning -ServiceControl.Audit Health: Audit retention \ No newline at end of file +ServiceControl.Audit Health: Audit retention +Storage space: ServiceControl body storage \ No newline at end of file diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/BodyIdPrefixTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/BodyIdPrefixTests.cs new file mode 100644 index 0000000000..f3054f1332 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/BodyIdPrefixTests.cs @@ -0,0 +1,28 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using Microsoft.Extensions.Logging.Abstractions; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + using ServiceControl.Audit.Persistence.EFCore.PostgreSql; + + [TestFixture] + class BodyIdPrefixTests + { + [Test] + public void A_dropped_partition_covers_the_bodies_of_its_whole_day() + { + var manager = new PostgreSqlAuditPartitionManager(NullLogger.Instance); + var day = new DateTime(2026, 10, 20, 0, 0, 0, DateTimeKind.Utc); + + var prefix = manager.BodyIdPrefix(day); + + using (Assert.EnterMultipleScope()) + { + Assert.That(prefix, Is.EqualTo("20261020")); + Assert.That(AuditBodyId.Format(day.AddHours(23), Guid.NewGuid()), Does.StartWith(prefix)); + Assert.That(AuditBodyId.Format(day.AddDays(1), Guid.NewGuid()), Does.Not.StartWith(prefix)); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PartlyExpiredDayRetentionTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PartlyExpiredDayRetentionTests.cs new file mode 100644 index 0000000000..b5884dd6c4 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PartlyExpiredDayRetentionTests.cs @@ -0,0 +1,47 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.IO; + using System.Threading.Tasks; + using NUnit.Framework; + using ServiceControl.Audit.Infrastructure; + + class PartlyExpiredDayRetentionTests : EFPersistenceTestFixture + { + public override Task Setup() + { + SetSettings = settings => + { + settings.AuditRetentionPeriod = TimeSpan.FromDays(1); + // FakeTimeProvider only moves forward; tomorrow's midday is independent of the current hour. + configuration.TimeProvider.SetUtcNow(new DateTimeOffset(configuration.TimeProvider.GetUtcNow().UtcDateTime.Date.AddDays(1).AddHours(12), TimeSpan.Zero)); + }; + return base.Setup(); + } + + [Test] + public async Task Keeps_the_body_and_row_until_the_whole_day_has_expired() + { + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + var retention = configuration.CreateRetention(); + + // The ingestion hour has expired but the rest of its day has not + configuration.TimeProvider.Advance(TimeSpan.FromHours(25)); + await retention.SweepNow(); + var partlyExpired = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + var filesWhilePartlyExpired = Directory.GetFiles(configuration.BodyStoragePath); + + configuration.TimeProvider.Advance(TimeSpan.FromHours(12)); + await retention.SweepNow(); + var expired = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(partlyExpired.Results, Has.Count.EqualTo(1)); + Assert.That(filesWhilePartlyExpired, Has.Length.EqualTo(1)); + Assert.That(expired.Results, Is.Empty); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath), Is.Empty); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs index e14534561d..5f4a45558f 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs @@ -14,6 +14,7 @@ namespace ServiceControl.Audit.Persistence.Tests using ServiceControl.Audit.Persistence.EFCore.DbContexts; using ServiceControl.Audit.Persistence.EFCore.Entities; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + using ServiceControl.Persistence.BodyStorage; class PostgreSqlEnvironmentDataTests : EFPersistenceTestFixture { @@ -66,6 +67,8 @@ public async Task Measures_and_identifies_the_search_path_schema_when_none_is_co var settings = new PersistenceSettings(TimeSpan.FromDays(1), true, 100000); settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.ConnectionStringKey] = new NpgsqlConnectionStringBuilder(configuration.ConnectionString) { SearchPath = configuration.Schema }.ConnectionString; + settings.PersisterSpecificSettings[BodyStorageSettingsParser.StorageTypeKey] = nameof(BodyStorageType.FileSystem); + settings.PersisterSpecificSettings[BodyStorageSettingsParser.FileSystemStoragePathKey] = configuration.BodyStoragePath; var services = new ServiceCollection(); services.AddLogging(); diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs index 5bc10040be..5a9b481a24 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs @@ -1,9 +1,11 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using NUnit.Framework; using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.PostgreSql; + using ServiceControl.Persistence.BodyStorage; [TestFixture] class RetentionPeriodTests @@ -26,6 +28,8 @@ static PersistenceSettings SettingsWith(TimeSpan retentionPeriod) { var settings = new PersistenceSettings(retentionPeriod, true, 100000); settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.ConnectionStringKey] = "Host=localhost"; + settings.PersisterSpecificSettings[BodyStorageSettingsParser.StorageTypeKey] = nameof(BodyStorageType.FileSystem); + settings.PersisterSpecificSettings[BodyStorageSettingsParser.FileSystemStoragePathKey] = Path.GetTempPath(); return settings; } } diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj index 1ea1c0eec9..35b1d895d3 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj @@ -36,8 +36,8 @@ - - + + diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt index cd5cf1e630..8b38d4eddb 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt @@ -1 +1,2 @@ -ServiceControl.Audit Health: Audit retention \ No newline at end of file +ServiceControl.Audit Health: Audit retention +Storage space: ServiceControl body storage \ No newline at end of file diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/BodyIdPrefixTests.cs b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/BodyIdPrefixTests.cs new file mode 100644 index 0000000000..3146f1589e --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/BodyIdPrefixTests.cs @@ -0,0 +1,26 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + using ServiceControl.Audit.Persistence.EFCore.SqlServer; + + [TestFixture] + class BodyIdPrefixTests + { + [Test] + public void An_expired_hour_covers_only_the_bodies_of_that_hour() + { + var hour = new DateTime(2026, 10, 20, 13, 0, 0, DateTimeKind.Utc); + + var prefix = new SqlServerAuditPartitionManager().BodyIdPrefix(hour); + + using (Assert.EnterMultipleScope()) + { + Assert.That(prefix, Is.EqualTo("2026102013-")); + Assert.That(AuditBodyId.Format(hour, Guid.NewGuid()), Does.StartWith(prefix)); + Assert.That(AuditBodyId.Format(hour.AddHours(1), Guid.NewGuid()), Does.Not.StartWith(prefix)); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj index b2a6fe42b5..d9c199f785 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj @@ -36,8 +36,8 @@ - - + + diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs index 1ce23546f8..b0d2aca71b 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs @@ -9,6 +9,7 @@ namespace ServiceControl.Audit.Persistence.Tests using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; using ServiceControl.Audit.Persistence.EFCore.SqlServer; + using ServiceControl.Persistence.BodyStorage; class SqlServerEnvironmentDataTests : EFPersistenceTestFixture { @@ -71,7 +72,7 @@ public void Should_report_only_the_service_objective_tier(string serviceObjectiv [TestCase("not a connection string")] public async Task Should_report_edition_and_service_objective_as_unknown_when_the_server_cannot_be_asked(string connectionString) { - var settings = new EFPersisterSettings { ConnectionString = connectionString, AuditRetentionPeriod = TimeSpan.FromDays(7), MaxBodySizeToStore = 1024 }; + var settings = new EFPersisterSettings { ConnectionString = connectionString, AuditRetentionPeriod = TimeSpan.FromDays(7), BodyStorage = new FileSystemBodyStorageSettings { StoragePath = "unused" } }; var probe = new SqlServerDatabaseHostingProbe(settings, new UnreachableScopeFactory(), NullLogger.Instance); var hosting = await probe.Probe(TestTimeoutCancellationToken); diff --git a/src/ServiceControl.Audit.Persistence.Tests/AuditTests.MaxBodySize.cs b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.MaxBodySize.cs new file mode 100644 index 0000000000..4a256f7833 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.MaxBodySize.cs @@ -0,0 +1,34 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Threading.Tasks; + using NUnit.Framework; + + partial class AuditTests + { + [Test] + public async Task Does_respect_max_message_body() + { + await using var unitOfWork = await StartAuditUnitOfWork(1); + + var body = new byte[MAX_BODY_SIZE + 1000]; + Random.Shared.NextBytes(body); + var processedMessage = MakeMessage(); + + await unitOfWork.RecordProcessedMessage(processedMessage, body); + + await unitOfWork.Complete(); + + var bodyId = GetBodyId(processedMessage); + + var retrievedMessage = await MessagesViewStore.GetMessageBody(bodyId, TestContext.CurrentContext.CancellationToken); + + Assert.That(retrievedMessage, Is.Not.Null); + using (Assert.EnterMultipleScope()) + { + Assert.That(retrievedMessage.Found, Is.True); + Assert.That(retrievedMessage.HasContent, Is.False); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs index 83532c48c4..1926be1dc7 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs @@ -100,31 +100,6 @@ public async Task Message_body_validator_is_stable_across_reads() Assert.That(second.Version, Is.EqualTo(first.Version), "the body did not change, so neither may its validator"); } - [Test] - public async Task Does_respect_max_message_body() - { - await using var unitOfWork = await StartAuditUnitOfWork(1); - - var body = new byte[MAX_BODY_SIZE + 1000]; - Random.Shared.NextBytes(body); - var processedMessage = MakeMessage(); - - await unitOfWork.RecordProcessedMessage(processedMessage, body); - - await unitOfWork.Complete(); - - var bodyId = GetBodyId(processedMessage); - - var retrievedMessage = await MessagesViewStore.GetMessageBody(bodyId, TestContext.CurrentContext.CancellationToken); - - Assert.That(retrievedMessage, Is.Not.Null); - using (Assert.EnterMultipleScope()) - { - Assert.That(retrievedMessage.Found, Is.True); - Assert.That(retrievedMessage.HasContent, Is.False); - } - } - [Test] public async Task Does_not_deduplicate_with_different_processing_started_header() { diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditBodyIdTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditBodyIdTests.cs new file mode 100644 index 0000000000..b427095bc8 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditBodyIdTests.cs @@ -0,0 +1,40 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + + [TestFixture] + class AuditBodyIdTests + { + static readonly DateTime Hour = new(2026, 10, 20, 13, 0, 0, DateTimeKind.Utc); + + [Test] + public void An_hour_prefix_matches_only_that_hour() + { + var prefix = AuditBodyId.HourPrefix(Hour); + + using (Assert.EnterMultipleScope()) + { + Assert.That(prefix, Is.EqualTo("2026102013-")); + Assert.That(AuditBodyId.Format(Hour, Guid.NewGuid()), Does.StartWith(prefix)); + Assert.That(AuditBodyId.Format(Hour.AddHours(1), Guid.NewGuid()), Does.Not.StartWith(prefix)); + } + } + + [Test] + public void A_day_prefix_matches_every_hour_of_that_day_and_no_other() + { + var prefix = AuditBodyId.DayPrefix(Hour.Date); + + using (Assert.EnterMultipleScope()) + { + Assert.That(prefix, Is.EqualTo("20261020")); + Assert.That(AuditBodyId.Format(Hour.Date, Guid.NewGuid()), Does.StartWith(prefix)); + Assert.That(AuditBodyId.Format(Hour.Date.AddHours(23), Guid.NewGuid()), Does.StartWith(prefix)); + Assert.That(AuditBodyId.Format(Hour.Date.AddDays(1), Guid.NewGuid()), Does.Not.StartWith(prefix)); + Assert.That(AuditBodyId.Format(Hour.Date.AddHours(-1), Guid.NewGuid()), Does.Not.StartWith(prefix)); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditIngestionUnitOfWorkBodyTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditIngestionUnitOfWorkBodyTests.cs new file mode 100644 index 0000000000..b049ee89d7 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditIngestionUnitOfWorkBodyTests.cs @@ -0,0 +1,142 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using System.Text; + using System.Threading.Tasks; + using Auditing; + using Microsoft.Extensions.DependencyInjection; + using NServiceBus; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Abstractions; + using ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + using ServiceControl.Persistence.BodyStorage; + + [TestFixture] + class AuditIngestionUnitOfWorkBodyTests + { + static readonly DateTime CreatedOn = new(2026, 10, 20, 13, 0, 0, DateTimeKind.Utc); + const int Max = 100; + + RecordingBodyStorage storage; + ThrowingScopeFactory scopes; + AuditIngestionUnitOfWork unitOfWork; + + [SetUp] + public void SetUp() + { + storage = new RecordingBodyStorage(); + scopes = new ThrowingScopeFactory(); + var settings = new EFPersisterSettings + { + ConnectionString = "unused", + AuditRetentionPeriod = TimeSpan.FromDays(1), + BodyStorage = new FileSystemBodyStorageSettings { StoragePath = "unused", MaxBodySizeToStore = Max } + }; + unitOfWork = new AuditIngestionUnitOfWork(scopes, storage, settings, CreatedOn); + } + + [TearDown] + public async Task TearDown() => await unitOfWork.DisposeAsync(); + + static ProcessedMessage Message(string contentType) + { + var headers = new Dictionary + { + [Headers.MessageId] = Guid.NewGuid().ToString(), + [Headers.ProcessingEndpoint] = "Receiver", + [Headers.ContentType] = contentType + }; + + return new ProcessedMessage(headers, []); + } + + [Test] + public async Task Small_text_is_not_written_externally() + { + await unitOfWork.RecordProcessedMessage(Message("application/json"), Encoding.UTF8.GetBytes("{}")); + + Assert.That(storage.Written, Is.Empty); + } + + [Test] + public async Task An_external_body_is_written_under_the_audit_body_id() + { + var message = Message("application/octet-stream"); + byte[] body = [1, 2, 3]; + + await unitOfWork.RecordProcessedMessage(message, body); + + using (Assert.EnterMultipleScope()) + { + Assert.That(storage.Written, Has.Count.EqualTo(1)); + Assert.That(storage.Written[0].BodyId, Is.EqualTo(message.Id)); + Assert.That(storage.Written[0].BodyId, Does.StartWith("2026102013-")); + Assert.That(storage.Written[0].Body, Is.EqualTo(body)); + Assert.That(storage.Written[0].ContentType, Is.EqualTo("application/octet-stream")); + } + } + + [Test] + public async Task An_oversized_text_body_is_written_in_full() + { + var body = Encoding.UTF8.GetBytes(new string('a', Max * 3)); + + await unitOfWork.RecordProcessedMessage(Message("text/plain"), body); + + Assert.That(storage.Written[0].Body, Is.EqualTo(body)); + } + + [Test] + public async Task A_failed_body_write_stops_the_batch_before_any_row_is_inserted() + { + storage.WriteFailure = new InvalidOperationException("storage is down"); + await unitOfWork.RecordProcessedMessage(Message("application/octet-stream"), new byte[] { 1 }); + + var exception = Assert.ThrowsAsync(() => unitOfWork.Complete()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(exception.Message, Is.EqualTo("storage is down")); + Assert.That(scopes.Used, Is.False, "the database must not be touched once a body write failed"); + } + } + + [Test] + public async Task Disposing_without_completing_waits_for_the_started_body_writes() + { + storage.PendingWrite = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await unitOfWork.RecordProcessedMessage(Message("application/octet-stream"), new byte[] { 1 }); + + var dispose = unitOfWork.DisposeAsync().AsTask(); + Assert.That(dispose.IsCompleted, Is.False); + + storage.PendingWrite.SetResult(); + await dispose; + } + + [Test] + public async Task Disposing_does_not_throw_when_a_started_body_write_failed() + { + storage.PendingWrite = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await unitOfWork.RecordProcessedMessage(Message("application/octet-stream"), new byte[] { 1 }); + + var dispose = unitOfWork.DisposeAsync().AsTask(); + storage.PendingWrite.SetException(new InvalidOperationException("storage is down")); + + Assert.DoesNotThrowAsync(async () => await dispose); + } + + sealed class ThrowingScopeFactory : IServiceScopeFactory + { + public bool Used { get; private set; } + + public IServiceScope CreateScope() + { + Used = true; + throw new InvalidOperationException("The database was used."); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/BodyStorageSettingsParserTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/BodyStorageSettingsParserTests.cs new file mode 100644 index 0000000000..e23716a8a8 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/BodyStorageSettingsParserTests.cs @@ -0,0 +1,188 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Abstractions; + using ServiceControl.Persistence.BodyStorage; + + [TestFixture] + class BodyStorageSettingsParserTests + { + const int MaxBodySizeToStore = 1234; + + static Dictionary Settings(params (string Key, string Value)[] values) + { + var settings = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var (key, value) in values) + { + settings[key] = value; + } + + return settings; + } + + static BodyStorageSettings Parse(params (string Key, string Value)[] values) => + BodyStorageSettingsParser.Parse(Settings(values), MaxBodySizeToStore); + + [Test] + public void The_storage_type_is_required() + { + var exception = Assert.Throws(() => Parse()); + + Assert.That(exception.Message, Does.Contain("MessageBody/StorageType")); + } + + [Test] + public void An_unknown_storage_type_is_rejected() => + Assert.Throws(() => Parse(("MessageBody/StorageType", "Tape"))); + + [Test] + public void File_system_settings_use_the_defaults() + { + var settings = (FileSystemBodyStorageSettings)Parse( + ("MessageBody/StorageType", "filesystem"), + ("MessageBody/FileSystem/StoragePath", "/bodies")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(settings.StoragePath, Is.EqualTo("/bodies")); + Assert.That(settings.DataSpaceRemainingThreshold, Is.EqualTo(FileSystemBodyStorageSettings.DefaultDataSpaceRemainingThreshold)); + Assert.That(settings.MinCompressionSize, Is.EqualTo(BodyStorageSettings.DefaultMinCompressionSize)); + Assert.That(settings.MaxBodySizeToStore, Is.EqualTo(MaxBodySizeToStore)); + } + } + + [Test] + public void The_file_system_path_is_required() => + Assert.Throws(() => Parse(("MessageBody/StorageType", "FileSystem"))); + + [TestCase("-1")] + [TestCase("101")] + [TestCase("lots")] + public void An_invalid_file_system_threshold_is_rejected(string threshold) => + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "FileSystem"), + ("MessageBody/FileSystem/StoragePath", "/bodies"), + ("MessageBody/FileSystem/DataSpaceRemainingThreshold", threshold))); + + [Test] + public void The_compression_threshold_can_be_set() + { + var settings = Parse( + ("MessageBody/StorageType", "FileSystem"), + ("MessageBody/FileSystem/StoragePath", "/bodies"), + ("MessageBody/MinCompressionSize", "10")); + + Assert.That(settings.MinCompressionSize, Is.EqualTo(10)); + } + + [Test] + public void An_invalid_max_body_size_falls_back_to_the_default() + { + var settings = BodyStorageSettingsParser.Parse( + Settings(("MessageBody/StorageType", "FileSystem"), ("MessageBody/FileSystem/StoragePath", "/bodies")), + 0); + + Assert.That(settings.MaxBodySizeToStore, Is.EqualTo(BodyStorageSettings.DefaultMaxBodySizeToStore)); + } + + [Test] + public void Azure_shared_key_uses_the_audit_container_by_default() + { + var settings = (AzureBlobBodyStorageSettings)Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ConnectionString", "UseDevelopmentStorage=true")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(settings.Authentication, Is.TypeOf()); + Assert.That(settings.ContainerName, Is.EqualTo("audit-bodies")); + } + } + + [Test] + public void Azure_managed_identity_reads_its_options() + { + var settings = (AzureBlobBodyStorageSettings)Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ServiceUri", "https://account.blob.core.windows.net"), + ("MessageBody/Azure/ManagedIdentityClientId", "client"), + ("MessageBody/Azure/ContainerName", "custom")); + + var authentication = (AzureBlobManagedIdentityAuthentication)settings.Authentication; + + using (Assert.EnterMultipleScope()) + { + Assert.That(authentication.ServiceUri, Is.EqualTo(new Uri("https://account.blob.core.windows.net"))); + Assert.That(authentication.ClientId, Is.EqualTo("client")); + Assert.That(settings.ContainerName, Is.EqualTo("custom")); + } + } + + [Test] + public void Azure_requires_exactly_one_authentication_mode() + { + Assert.Throws(() => Parse(("MessageBody/StorageType", "AzureBlob"))); + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ConnectionString", "UseDevelopmentStorage=true"), + ("MessageBody/Azure/ServiceUri", "https://account.blob.core.windows.net"))); + } + + [Test] + public void Azure_rejects_a_relative_service_uri() => + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ServiceUri", "account"))); + + [Test] + public void S3_uses_the_audit_prefix_by_default() + { + var settings = (S3BodyStorageSettings)Parse( + ("MessageBody/StorageType", "S3"), + ("MessageBody/S3/BucketName", "bucket")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(settings.BucketName, Is.EqualTo("bucket")); + Assert.That(settings.KeyPrefix, Is.EqualTo("audit-bodies/")); + Assert.That(settings.Credentials, Is.Null); + } + } + + [Test] + public void S3_requires_a_bucket() => + Assert.Throws(() => Parse(("MessageBody/StorageType", "S3"))); + + [Test] + public void S3_credentials_must_come_as_a_pair() + { + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "S3"), + ("MessageBody/S3/BucketName", "bucket"), + ("MessageBody/S3/AccessKeyId", "key"))); + + var settings = (S3BodyStorageSettings)Parse( + ("MessageBody/StorageType", "S3"), + ("MessageBody/S3/BucketName", "bucket"), + ("MessageBody/S3/AccessKeyId", "key"), + ("MessageBody/S3/SecretAccessKey", "secret")); + + Assert.That(settings.Credentials.AccessKeyId, Is.EqualTo("key")); + } + + [Test] + public void Every_key_is_read_from_the_configuration() + { + Assert.That(new TestConfiguration().ConfigurationKeys, Is.SupersetOf(BodyStorageSettingsParser.Keys)); + } + + sealed class TestConfiguration : EFPersistenceConfigurationBase + { + public override string Name => "Test"; + + protected override IPersistence Create(EFPersisterSettings settings) => throw new NotSupportedException(); + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs index a25ca35b7e..67283fda28 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs @@ -1,11 +1,14 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using System.Linq; using System.Text; using System.Threading.Tasks; + using Microsoft.EntityFrameworkCore; using NUnit.Framework; using ServiceControl.Audit.Infrastructure; + using ServiceControl.Audit.Persistence.EFCore.Entities; class EFBodyTests : EFPersistenceTestFixture { @@ -44,7 +47,15 @@ public async Task Finds_a_body_by_the_unique_message_id_alone() } [Test] - public async Task A_text_body_over_the_limit_is_not_served_but_stays_searchable() + public async Task A_small_text_body_is_kept_inline() + { + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes("text")); + + Assert.That(Directory.GetFiles(configuration.BodyStoragePath), Is.Empty); + } + + [Test] + public async Task A_text_body_over_the_limit_is_served_in_full_and_stays_searchable() { var text = "findme " + new string('x', MaxBodySizeToStore); await Ingest(MakeMessage(), Encoding.UTF8.GetBytes(text)); @@ -54,36 +65,63 @@ public async Task A_text_body_over_the_limit_is_not_served_but_stays_searchable( using (Assert.EnterMultipleScope()) { - Assert.That(body.Found, Is.True); - Assert.That(body.HasContent, Is.False, "the body is over MaxBodySizeToStore, which answers 204"); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath), Has.Length.EqualTo(1)); + Assert.That(await Read(body), Is.EqualTo(Encoding.UTF8.GetBytes(text))); + Assert.That(body.ContentType, Is.EqualTo("application/json")); + Assert.That(body.ContentLength, Is.EqualTo(text.Length)); + Assert.That(body.Version.HasValue, Is.True); Assert.That(found.Results, Has.Count.EqualTo(1)); } } [Test] - public async Task A_binary_body_is_not_stored() + public async Task A_binary_body_is_served_with_its_exact_bytes() { - await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + byte[] bytes = [0, 1, 2, 255, 254]; + await Ingest(MakeMessage(contentType: "application/octet-stream"), bytes); var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); - Assert.That(body.Found, Is.False); + using (Assert.EnterMultipleScope()) + { + Assert.That(body.Found, Is.True); + Assert.That(body.HasContent, Is.True); + Assert.That(body.ContentType, Is.EqualTo("application/octet-stream")); + Assert.That(body.ContentLength, Is.EqualTo(bytes.Length)); + Assert.That(await Read(body), Is.EqualTo(bytes)); + } } [Test] - public async Task A_body_that_is_not_utf8_is_not_stored() + public async Task A_body_that_is_not_utf8_is_served_with_its_exact_bytes() { - await Ingest(MakeMessage(contentType: null), [0xC3, 0x28]); + byte[] bytes = [0xC3, 0x28]; + await Ingest(MakeMessage(contentType: null), bytes); var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); - Assert.That(body.Found, Is.False); + Assert.That(await Read(body), Is.EqualTo(bytes)); + } + + [Test] + public async Task A_text_body_with_a_nul_is_served_with_its_exact_bytes() + { + byte[] bytes = "a\0b"u8.ToArray(); + await Ingest(MakeMessage(), bytes); + + var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); + + Assert.That(await Read(body), Is.EqualTo(bytes)); } [Test] - public async Task A_binary_body_over_the_limit_answers_that_it_was_too_large() + public async Task A_missing_external_body_answers_no_content() { - await Ingest(MakeMessage(contentType: "application/octet-stream"), new byte[MaxBodySizeToStore + 1]); + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + foreach (var file in Directory.GetFiles(configuration.BodyStoragePath)) + { + File.Delete(file); + } var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); @@ -94,6 +132,50 @@ public async Task A_binary_body_over_the_limit_answers_that_it_was_too_large() } } + [Test] + public async Task A_row_written_before_external_storage_that_was_too_large_answers_no_content() + { + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes("text")); + await SetBodyState(BodyState.TooLarge); + + var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(body.Found, Is.True); + Assert.That(body.HasContent, Is.False); + } + } + + [Test] + public async Task A_row_written_before_external_storage_that_was_not_text_is_not_found() + { + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes("text")); + await SetBodyState(BodyState.NotText); + + var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); + + Assert.That(body.Found, Is.False); + } + + Task SetBodyState(BodyState state) => + WithDbContext((dbContext, token) => dbContext.AuditMessages.ExecuteUpdateAsync(setters => setters.SetProperty(message => message.BodyState, state), token)); + + static async Task Read(MessageBodyView body) + { + if (body.StreamContent == null) + { + return Encoding.UTF8.GetBytes(body.StringContent); + } + + await using (body.StreamContent) + { + using var buffer = new MemoryStream(); + await body.StreamContent.CopyToAsync(buffer); + return buffer.ToArray(); + } + } + [Test] public async Task A_message_without_a_body_has_no_body_url() { diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs index be78469c28..89e2bcefd0 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using System.Linq; using System.Threading; using System.Threading.Tasks; @@ -9,8 +10,10 @@ namespace ServiceControl.Audit.Persistence.Tests using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Time.Testing; using NServiceBus.CustomChecks; + using NUnit.Framework; using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.Implementation; + using ServiceControl.Persistence.BodyStorage; using UnitOfWork; abstract class EFPersistenceTestsConfiguration : IPersistenceTestsConfiguration @@ -31,6 +34,8 @@ abstract class EFPersistenceTestsConfiguration : IPersistenceTestsConfiguration public string ConnectionString { get; private set; } + public string BodyStoragePath { get; } = Directory.CreateTempSubdirectory("sc_audit_bodies_").FullName; + public string Schema { get; } = $"sc_test_{Guid.NewGuid():N}"; // PostgreSQL keeps only microseconds, so a finer seed would not round trip. @@ -44,6 +49,8 @@ public async Task Configure(Action setSettings) var settings = new PersistenceSettings(TimeSpan.FromDays(1), true, 100000); settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.ConnectionStringKey] = ConnectionString; settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.SchemaKey] = Schema; + settings.PersisterSpecificSettings[BodyStorageSettingsParser.StorageTypeKey] = nameof(BodyStorageType.FileSystem); + settings.PersisterSpecificSettings[BodyStorageSettingsParser.FileSystemStoragePathKey] = BodyStoragePath; setSettings(settings); @@ -70,6 +77,12 @@ public async Task Configure(Action setSettings) } } + // Registered on the endpoint in production, which these tests do not build. + if (hostBuilder.Services.Any(descriptor => descriptor.ServiceType == typeof(FileSystemBodyStorageSettings))) + { + hostBuilder.Services.AddTransient(); + } + host = hostBuilder.Build(); await host.StartAsync(); @@ -96,9 +109,21 @@ public async Task Cleanup() { await DropSchema(ConnectionString, Schema); } + + try + { + Directory.Delete(BodyStoragePath, recursive: true); + } + catch (DirectoryNotFoundException) + { + } + catch (IOException ex) + { + await TestContext.Out.WriteLineAsync($"WARNING: Could not delete body storage '{BodyStoragePath}': {ex.Message}"); + } } - internal AuditRetention CreateRetention() => ActivatorUtilities.CreateInstance(ServiceProvider); + internal AuditRetention CreateRetention(params object[] overrides) => ActivatorUtilities.CreateInstance(ServiceProvider, overrides); public abstract Task GetConnectionString(); diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs index fa342f4a46..5452bd2ee3 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs @@ -111,6 +111,8 @@ public Task> ListExpired(AuditDbContext dbContext, DateT public Task DropExpired(AuditDbContext dbContext, DateTime start, int batchSize, CancellationToken cancellationToken = default) => Task.FromResult(new DropResult(RowsDeleted: 0, Completed: true, Deferred: true)); + public string BodyIdPrefix(DateTime expired) => AuditBodyId.HourPrefix(expired); + public Task ProvisionedUntil(AuditDbContext dbContext, CancellationToken cancellationToken = default) => Task.FromResult(null); } diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs index d06a996705..3d13e1d043 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using System.Linq; using System.Threading.Tasks; using Microsoft.Extensions.DependencyInjection; @@ -36,6 +37,58 @@ public async Task Removes_rows_older_than_the_retention_period() Assert.That(remaining.Results.Select(message => message.MessageId), Is.EqualTo(new[] { live.Headers[Headers.MessageId] })); } + [Test] + public async Task Removes_the_external_bodies_with_the_rows() + { + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath), Has.Length.EqualTo(1)); + + configuration.TimeProvider.Advance(TimeSpan.FromDays(3)); + await configuration.CreateRetention().SweepNow(); + + var remaining = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(remaining.Results, Is.Empty); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath), Is.Empty); + } + } + + [Test] + public async Task Deletes_the_bodies_by_the_prefix_of_the_expired_rows() + { + var message = MakeMessage(contentType: "application/octet-stream"); + await Ingest(message, [1, 2, 3]); + var bodyId = (await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc"))).Results.Single().BodyUrl.Split('/')[2]; + + configuration.TimeProvider.Advance(TimeSpan.FromDays(3)); + var storage = new RecordingBodyStorage(); + await configuration.CreateRetention(storage).SweepNow(); + + Assert.That(storage.DeletedPrefixes, Has.Some.Matches(bodyId.StartsWith)); + } + + [Test] + public async Task Keeps_the_rows_when_deleting_their_bodies_fails() + { + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + + configuration.TimeProvider.Advance(TimeSpan.FromDays(3)); + var storage = new RecordingBodyStorage { DeleteFailure = new IOException("storage is down") }; + + Assert.ThrowsAsync(() => configuration.CreateRetention(storage).SweepNow()); + + var remaining = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + var body = await MessagesViewStore.GetMessageBody(remaining.Results.Single().BodyUrl.Split('/')[2], TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(remaining.Results, Has.Count.EqualTo(1)); + Assert.That(body.HasContent, Is.True, "the body is still in storage"); + } + } + [Test] public async Task Keeps_rows_until_the_retention_period_has_passed() { diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/MessageBodyClassifierTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/MessageBodyClassifierTests.cs new file mode 100644 index 0000000000..20354546da --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/MessageBodyClassifierTests.cs @@ -0,0 +1,124 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using System.Text; + using NServiceBus; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Entities; + using ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; + + [TestFixture] + class MessageBodyClassifierTests + { + const int Max = 100; + + static Dictionary Json => new() { [Headers.ContentType] = "application/json" }; + + [Test] + public void An_empty_body_has_no_body() + { + var (state, text) = MessageBodyClassifier.Classify(Json, ReadOnlyMemory.Empty, Max); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.None)); + Assert.That(text, Is.Null); + } + } + + [Test] + public void Small_text_stays_inline() + { + var (state, text) = MessageBodyClassifier.Classify(Json, Encoding.UTF8.GetBytes("{}"), Max); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.Stored)); + Assert.That(text, Is.EqualTo("{}")); + } + } + + [Test] + public void Text_at_the_limit_stays_inline() + { + var (state, _) = MessageBodyClassifier.Classify(Json, Encoding.UTF8.GetBytes(new string('a', Max)), Max); + + Assert.That(state, Is.EqualTo(BodyState.Stored)); + } + + [Test] + public void Text_over_the_limit_goes_external_and_keeps_a_prefix() + { + var (state, text) = MessageBodyClassifier.Classify(Json, Encoding.UTF8.GetBytes(new string('a', Max + 1)), Max); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.EqualTo(new string('a', Max))); + } + } + + [Test] + public void The_prefix_does_not_split_a_multi_byte_character() + { + // 99 ASCII bytes then a 3 byte character that straddles the limit + var body = Encoding.UTF8.GetBytes(new string('a', Max - 1) + "€"); + + var (state, text) = MessageBodyClassifier.Classify(Json, body, Max); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.EqualTo(new string('a', Max - 1))); + } + } + + [TestCase("application/octet-stream")] + [TestCase("application/json; binary")] + public void Binary_content_goes_external_without_a_prefix(string contentType) + { + var (state, text) = MessageBodyClassifier.Classify(new Dictionary { [Headers.ContentType] = contentType }, new byte[] { 1, 2, 3 }, Max); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.Null); + } + } + + [Test] + public void Encoded_content_goes_external() + { + var headers = new Dictionary { ["Content-Encoding"] = "gzip" }; + + var (state, _) = MessageBodyClassifier.Classify(headers, Encoding.UTF8.GetBytes("text"), Max); + + Assert.That(state, Is.EqualTo(BodyState.External)); + } + + [Test] + public void A_body_that_is_not_utf8_goes_external() + { + var (state, text) = MessageBodyClassifier.Classify(new Dictionary(), new byte[] { 0xC3, 0x28 }, Max); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.Null); + } + } + + [Test] + public void A_body_with_a_nul_goes_external() + { + var (state, text) = MessageBodyClassifier.Classify(Json, "a\0b"u8.ToArray(), Max); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.Null); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/RecordingBodyStorage.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/RecordingBodyStorage.cs new file mode 100644 index 0000000000..e65516fb6d --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/RecordingBodyStorage.cs @@ -0,0 +1,56 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using System.Linq; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Persistence.BodyStorage; + + class RecordingBodyStorage : IBodyStoragePersistence + { + public List<(string BodyId, byte[] Body, string ContentType)> Written { get; } = []; + + public List DeletedPrefixes { get; } = []; + + public Exception WriteFailure { get; set; } + + public Exception DeleteFailure { get; set; } + + public TaskCompletionSource PendingWrite { get; set; } + + public Task WriteBody(string bodyId, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default) + { + if (PendingWrite != null) + { + return PendingWrite.Task; + } + + if (WriteFailure != null) + { + return Task.FromException(WriteFailure); + } + + Written.Add((bodyId, body.ToArray(), contentType)); + return Task.CompletedTask; + } + + public Task ReadBody(string bodyId, CancellationToken cancellationToken = default) => + Task.FromResult(null); + + public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default) => Task.CompletedTask; + + public Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + if (DeleteFailure != null) + { + return Task.FromException(DeleteFailure); + } + + DeletedPrefixes.Add(prefix); + return Task.CompletedTask; + } + + public string[] WrittenIds => [.. Written.Select(write => write.BodyId)]; + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/SchemaMustExistTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/SchemaMustExistTests.cs index 6ef1503937..fea4075e5b 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/SchemaMustExistTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/SchemaMustExistTests.cs @@ -6,6 +6,7 @@ namespace ServiceControl.Audit.Persistence.Tests using Microsoft.Extensions.Hosting; using NUnit.Framework; using ServiceControl.Audit.Persistence.EFCore.Abstractions; + using ServiceControl.Persistence.BodyStorage; [TestFixture] class SchemaMustExistTests @@ -19,14 +20,23 @@ public async Task Setup_fails_when_the_configured_schema_does_not_exist() var settings = new PersistenceSettings(TimeSpan.FromDays(1), true, 100000); settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.ConnectionStringKey] = await testsConfiguration.GetConnectionString(); settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.SchemaKey] = schema; + settings.PersisterSpecificSettings[BodyStorageSettingsParser.StorageTypeKey] = nameof(BodyStorageType.FileSystem); + settings.PersisterSpecificSettings[BodyStorageSettingsParser.FileSystemStoragePathKey] = testsConfiguration.BodyStoragePath; - var hostBuilder = Host.CreateApplicationBuilder(); - testsConfiguration.CreateConfiguration().Create(settings).AddInstaller(hostBuilder.Services); - using var host = hostBuilder.Build(); + try + { + var hostBuilder = Host.CreateApplicationBuilder(); + testsConfiguration.CreateConfiguration().Create(settings).AddInstaller(hostBuilder.Services); + using var host = hostBuilder.Build(); - var exception = Assert.ThrowsAsync(() => host.StartAsync()); + var exception = Assert.ThrowsAsync(() => host.StartAsync()); - Assert.That(exception.Message, Does.Contain(schema).And.Contain("does not exist")); + Assert.That(exception.Message, Does.Contain(schema).And.Contain("does not exist")); + } + finally + { + await testsConfiguration.Cleanup(); + } } } } diff --git a/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs b/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs index af61ccd47a..3e38d8a7a2 100644 --- a/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs +++ b/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs @@ -48,6 +48,7 @@ public async Task Should_provision_queues() var settings = new Settings("FakeTransport", "FakePersistence") { InstanceName = instanceInputQueueName, + IngestAuditMessages = true, ForwardAuditMessages = true, AssemblyLoadContextResolver = static _ => AssemblyLoadContext.Default }; diff --git a/src/ServiceControl.Audit/App.config b/src/ServiceControl.Audit/App.config index 5ea329b21c..18f35bd6c5 100644 --- a/src/ServiceControl.Audit/App.config +++ b/src/ServiceControl.Audit/App.config @@ -23,6 +23,8 @@ These settings are only here so that we can debug ServiceControl while developin + + diff --git a/src/ServiceControl.Infrastructure.Tests/BodyStorage/FileSystemBodyStoragePersistenceTests.cs b/src/ServiceControl.Infrastructure.Tests/BodyStorage/FileSystemBodyStoragePersistenceTests.cs new file mode 100644 index 0000000000..f3f9e12cca --- /dev/null +++ b/src/ServiceControl.Infrastructure.Tests/BodyStorage/FileSystemBodyStoragePersistenceTests.cs @@ -0,0 +1,162 @@ +namespace ServiceControl.Infrastructure.Tests.BodyStorage; + +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Persistence.BodyStorage; + +[TestFixture] +class FileSystemBodyStoragePersistenceTests +{ + string tempDir; + FileSystemBodyStoragePersistence store; + + [SetUp] + public void SetUp() + { + tempDir = Directory.CreateTempSubdirectory("sc-body-tests-").FullName; + store = new FileSystemBodyStoragePersistence(new FileSystemBodyStorageSettings { StoragePath = tempDir, MinCompressionSize = 64 }); + } + + [TearDown] + public void TearDown() => Directory.Delete(tempDir, recursive: true); + + [TestCase(11)] + [TestCase(100_000)] + public async Task Round_trips_compressed_and_uncompressed_bodies(int bodyLength) + { + var body = Encoding.UTF8.GetBytes(new string('a', bodyLength)); + await store.WriteBody("body-1", body, "application/json"); + + var result = await store.ReadBody("body-1"); + + Assert.That(result, Is.Not.Null); + using (result.Stream) + { + Assert.Multiple(() => + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + Assert.That(result.ContentType, Is.EqualTo("application/json")); + Assert.That(result.BodySize, Is.EqualTo(bodyLength)); + Assert.That(result.Stream.CanSeek, Is.False, "a seekable stream would advertise the header bytes in its Length"); + }); + } + } + + [Test] + public async Task Rewriting_a_body_keeps_the_first_write() + { + await store.WriteBody("body-1", Encoding.UTF8.GetBytes("original"), "text/plain"); + await store.WriteBody("body-1", Encoding.UTF8.GetBytes("different"), "text/plain"); + + var result = await store.ReadBody("body-1"); + + using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(Encoding.UTF8.GetBytes("original"))); + } + } + + [Test] + public async Task Missing_body_reads_as_null_and_deletes_quietly() + { + Assert.That(await store.ReadBody("missing"), Is.Null); + Assert.DoesNotThrowAsync(() => store.DeleteBodyIfExists("missing")); + } + + [Test] + public async Task Delete_with_prefix_removes_only_bodies_with_that_prefix() + { + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("2026102010-a", body, "text/plain"); + await store.WriteBody("2026102010-b", body, "text/plain"); + await store.WriteBody("2026102011-a", body, "text/plain"); + + await store.DeleteBodiesWithPrefix("2026102010"); + + Assert.That(await store.ReadBody("2026102010-a"), Is.Null); + Assert.That(await store.ReadBody("2026102010-b"), Is.Null); + Assert.That(File.Exists(Path.Combine(tempDir, "2026102011-a.body")), Is.True); + } + + [Test] + public async Task Delete_with_prefix_leaves_non_body_files() + { + var other = Path.Combine(tempDir, $"2026102010-a.body.{Guid.NewGuid():N}.tmp"); + await File.WriteAllTextAsync(other, "x"); + + await store.DeleteBodiesWithPrefix("2026102010"); + + Assert.That(File.Exists(other), Is.True); + } + + [TestCase("prefix", "prefix-a", "PREFIX-b")] + [TestCase("<", "keep-a", "keep-b")] + public async Task Delete_with_prefix_uses_literal_ordinal_matching(string prefix, string firstId, string secondId) + { + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody(firstId, body, "text/plain"); + await store.WriteBody(secondId, body, "text/plain"); + + await store.DeleteBodiesWithPrefix(prefix); + + Assert.That(File.Exists(Path.Combine(tempDir, $"{firstId}.body")), Is.EqualTo(!firstId.StartsWith(prefix, StringComparison.Ordinal))); + Assert.That(File.Exists(Path.Combine(tempDir, $"{secondId}.body")), Is.True); + } + + [Test] + public void Delete_with_prefix_succeeds_when_the_storage_directory_is_missing() + { + var missing = new FileSystemBodyStoragePersistence(new FileSystemBodyStorageSettings { StoragePath = Path.Combine(tempDir, "nope") }); + + Assert.DoesNotThrowAsync(() => missing.DeleteBodiesWithPrefix("2026102010")); + } + + [TestCase("")] + [TestCase(" ")] + [TestCase("*")] + [TestCase("a?")] + [TestCase("../x")] + [TestCase("a/b")] + [TestCase("a\\b")] + public async Task Delete_with_prefix_rejects_prefixes_that_could_widen_the_match(string prefix) + { + await store.WriteBody("keep", Encoding.UTF8.GetBytes("payload"), "text/plain"); + + Assert.ThrowsAsync(() => store.DeleteBodiesWithPrefix(prefix)); + Assert.That(File.Exists(Path.Combine(tempDir, "keep.body")), Is.True); + } + + [Test] + public async Task Delete_with_prefix_honors_cancellation() + { + await store.WriteBody("a-1", Encoding.UTF8.GetBytes("payload"), "text/plain"); + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + + Assert.ThrowsAsync(() => store.DeleteBodiesWithPrefix("a", cts.Token)); + Assert.That(Directory.EnumerateFiles(tempDir).Count(), Is.EqualTo(1)); + } + + [Test] + public async Task Provisioning_creates_the_storage_directory() + { + var path = Path.Combine(tempDir, "nested", "bodies"); + + await BodyStorageProvisioner.Provision(new FileSystemBodyStorageSettings { StoragePath = path }); + await BodyStorageProvisioner.Provision(new FileSystemBodyStorageSettings { StoragePath = path }); + + Assert.That(Directory.Exists(path), Is.True); + } + + static byte[] ReadAll(Stream stream) + { + using var buffer = new MemoryStream(); + stream.CopyTo(buffer); + return buffer.ToArray(); + } +} diff --git a/src/ServiceControl.Infrastructure.Tests/ServiceControl.Infrastructure.Tests.csproj b/src/ServiceControl.Infrastructure.Tests/ServiceControl.Infrastructure.Tests.csproj index 147347f072..f1c4fa7d61 100644 --- a/src/ServiceControl.Infrastructure.Tests/ServiceControl.Infrastructure.Tests.csproj +++ b/src/ServiceControl.Infrastructure.Tests/ServiceControl.Infrastructure.Tests.csproj @@ -7,6 +7,7 @@ + diff --git a/src/ServiceControl.Persistence.BodyStorage/.editorconfig b/src/ServiceControl.Persistence.BodyStorage/.editorconfig new file mode 100644 index 0000000000..ff993b49bb --- /dev/null +++ b/src/ServiceControl.Persistence.BodyStorage/.editorconfig @@ -0,0 +1,4 @@ +[*.cs] + +# Justification: ServiceControl app has no synchronization context +dotnet_diagnostic.CA2007.severity = none diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs b/src/ServiceControl.Persistence.BodyStorage/AzureBlobBodyStoragePersistence.cs similarity index 88% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs rename to src/ServiceControl.Persistence.BodyStorage/AzureBlobBodyStoragePersistence.cs index f07ab437a7..38c7e9a6b6 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.BodyStorage/AzureBlobBodyStoragePersistence.cs @@ -1,12 +1,10 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; using System.IO.Compression; using System.Net; using Azure; using Azure.Storage.Blobs; using Azure.Storage.Blobs.Models; -using ServiceControl.Persistence.EFCore.Abstractions; -using ServiceControl.Persistence.EFCore.Infrastructure; public class AzureBlobBodyStoragePersistence : IBodyStoragePersistence { @@ -95,4 +93,14 @@ public async Task WriteBody(string bodyId, ReadOnlyMemory body, string con public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default) => container.GetBlobClient(bodyId).DeleteIfExistsAsync(cancellationToken: cancellationToken); + + public async Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + BodyIdPrefix.Validate(prefix); + + await foreach (var blob in container.GetBlobsAsync(BlobTraits.None, BlobStates.None, prefix, cancellationToken: cancellationToken)) + { + await container.GetBlobClient(blob.Name).DeleteIfExistsAsync(cancellationToken: cancellationToken); + } + } } diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/AzureBlobBodyStorageSettings.cs b/src/ServiceControl.Persistence.BodyStorage/AzureBlobBodyStorageSettings.cs similarity index 94% rename from src/ServiceControl.Persistence.EFCore/Abstractions/AzureBlobBodyStorageSettings.cs rename to src/ServiceControl.Persistence.BodyStorage/AzureBlobBodyStorageSettings.cs index 0d7aec07ac..a000458f5f 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/AzureBlobBodyStorageSettings.cs +++ b/src/ServiceControl.Persistence.BodyStorage/AzureBlobBodyStorageSettings.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Abstractions; +namespace ServiceControl.Persistence.BodyStorage; public sealed class AzureBlobBodyStorageSettings : BodyStorageSettings { diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobClientFactory.cs b/src/ServiceControl.Persistence.BodyStorage/AzureBlobClientFactory.cs similarity index 91% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobClientFactory.cs rename to src/ServiceControl.Persistence.BodyStorage/AzureBlobClientFactory.cs index 6c2ed6308c..9bb5afc532 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobClientFactory.cs +++ b/src/ServiceControl.Persistence.BodyStorage/AzureBlobClientFactory.cs @@ -1,9 +1,8 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; using Azure.Core; using Azure.Identity; using Azure.Storage.Blobs; -using ServiceControl.Persistence.EFCore.Abstractions; static class AzureBlobClientFactory { diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyCompression.cs b/src/ServiceControl.Persistence.BodyStorage/BodyCompression.cs similarity index 95% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyCompression.cs rename to src/ServiceControl.Persistence.BodyStorage/BodyCompression.cs index 2946391dec..8943b413b6 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyCompression.cs +++ b/src/ServiceControl.Persistence.BodyStorage/BodyCompression.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; using System.Buffers; using System.IO.Compression; diff --git a/src/ServiceControl.Persistence.BodyStorage/BodyIdPrefix.cs b/src/ServiceControl.Persistence.BodyStorage/BodyIdPrefix.cs new file mode 100644 index 0000000000..118fc89447 --- /dev/null +++ b/src/ServiceControl.Persistence.BodyStorage/BodyIdPrefix.cs @@ -0,0 +1,17 @@ +namespace ServiceControl.Persistence.BodyStorage; + +static class BodyIdPrefix +{ + static readonly char[] Forbidden = ['*', '?', '/', '\\']; + + // A blank prefix would match every body, and wildcard or path characters would widen or escape the match. + public static void Validate(string prefix) + { + ArgumentException.ThrowIfNullOrWhiteSpace(prefix); + + if (prefix.IndexOfAny(Forbidden) >= 0 || prefix.Contains("..")) + { + throw new ArgumentException($"Body id prefix '{prefix}' must not contain wildcard or path characters.", nameof(prefix)); + } + } +} diff --git a/src/ServiceControl.Persistence.BodyStorage/BodyStorageProvisioner.cs b/src/ServiceControl.Persistence.BodyStorage/BodyStorageProvisioner.cs new file mode 100644 index 0000000000..117baac693 --- /dev/null +++ b/src/ServiceControl.Persistence.BodyStorage/BodyStorageProvisioner.cs @@ -0,0 +1,37 @@ +namespace ServiceControl.Persistence.BodyStorage; + +using Amazon.S3.Model; +using Amazon.S3.Util; + +/// +/// Creates the directory, container or bucket that the selected body storage type writes to. +/// +public static class BodyStorageProvisioner +{ + public static Task Provision(BodyStorageSettings settings, CancellationToken cancellationToken = default) => + settings switch + { + FileSystemBodyStorageSettings fileSystem => ProvisionFileSystem(fileSystem, cancellationToken), + AzureBlobBodyStorageSettings azureBlob => AzureBlobClientFactory.CreateContainerClient(azureBlob).CreateIfNotExistsAsync(cancellationToken: cancellationToken), + S3BodyStorageSettings s3 => ProvisionS3(s3, cancellationToken), + _ => throw new ArgumentOutOfRangeException(nameof(settings), settings, "Unknown body storage type.") + }; + + static Task ProvisionFileSystem(FileSystemBodyStorageSettings settings, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + Directory.CreateDirectory(settings.StoragePath); + + return Task.CompletedTask; + } + + static async Task ProvisionS3(S3BodyStorageSettings settings, CancellationToken cancellationToken) + { + using var client = S3ClientFactory.Create(settings); + + if (!await AmazonS3Util.DoesS3BucketExistV2Async(client, settings.BucketName)) + { + await client.PutBucketAsync(new PutBucketRequest { BucketName = settings.BucketName }, cancellationToken); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/BodyStorageSettings.cs b/src/ServiceControl.Persistence.BodyStorage/BodyStorageSettings.cs similarity index 91% rename from src/ServiceControl.Persistence.EFCore/Abstractions/BodyStorageSettings.cs rename to src/ServiceControl.Persistence.BodyStorage/BodyStorageSettings.cs index dd7ecb9e67..5088e96cd3 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/BodyStorageSettings.cs +++ b/src/ServiceControl.Persistence.BodyStorage/BodyStorageSettings.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Abstractions; +namespace ServiceControl.Persistence.BodyStorage; /// /// Settings for the selected body storage type. diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/BodyStorageType.cs b/src/ServiceControl.Persistence.BodyStorage/BodyStorageType.cs similarity index 55% rename from src/ServiceControl.Persistence.EFCore/Abstractions/BodyStorageType.cs rename to src/ServiceControl.Persistence.BodyStorage/BodyStorageType.cs index 9485cab036..cc967e752e 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/BodyStorageType.cs +++ b/src/ServiceControl.Persistence.BodyStorage/BodyStorageType.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Abstractions; +namespace ServiceControl.Persistence.BodyStorage; public enum BodyStorageType { diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/DriveInfoSpaceProvider.cs b/src/ServiceControl.Persistence.BodyStorage/DriveInfoSpaceProvider.cs similarity index 79% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/DriveInfoSpaceProvider.cs rename to src/ServiceControl.Persistence.BodyStorage/DriveInfoSpaceProvider.cs index 3cdff0f71b..703735c0d9 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/DriveInfoSpaceProvider.cs +++ b/src/ServiceControl.Persistence.BodyStorage/DriveInfoSpaceProvider.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; public class DriveInfoSpaceProvider : IDriveSpaceProvider { diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/ExpectedLengthStream.cs b/src/ServiceControl.Persistence.BodyStorage/ExpectedLengthStream.cs similarity index 97% rename from src/ServiceControl.Persistence.EFCore/Infrastructure/ExpectedLengthStream.cs rename to src/ServiceControl.Persistence.BodyStorage/ExpectedLengthStream.cs index 5d09f35b37..cd6a0dab89 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/ExpectedLengthStream.cs +++ b/src/ServiceControl.Persistence.BodyStorage/ExpectedLengthStream.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Infrastructure; +namespace ServiceControl.Persistence.BodyStorage; /// /// Validates the byte count of a forward-only stream without buffering its contents. diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageCustomCheck.cs b/src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStorageCustomCheck.cs similarity index 95% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageCustomCheck.cs rename to src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStorageCustomCheck.cs index 425c2f5140..994e336416 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageCustomCheck.cs +++ b/src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStorageCustomCheck.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; using System; using System.IO; @@ -6,7 +6,6 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; using System.Threading.Tasks; using Microsoft.Extensions.Logging; using NServiceBus.CustomChecks; -using ServiceControl.Persistence.EFCore.Abstractions; public class FileSystemBodyStorageCustomCheck(FileSystemBodyStorageSettings settings, IDriveSpaceProvider driveSpaceProvider, ILogger logger) : CustomCheck("ServiceControl body storage", "Storage space", TimeSpan.FromMinutes(15)) diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs b/src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStoragePersistence.cs similarity index 88% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs rename to src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStoragePersistence.cs index a9ed66165b..5ae2306305 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStoragePersistence.cs @@ -1,9 +1,7 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; using System.IO.Compression; using System.Text; -using ServiceControl.Persistence.EFCore.Abstractions; -using ServiceControl.Persistence.EFCore.Infrastructure; /// /// Stores message bodies as files, each one a small header followed by the body. @@ -141,6 +139,28 @@ public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToke return Task.CompletedTask; } + public Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + BodyIdPrefix.Validate(prefix); + cancellationToken.ThrowIfCancellationRequested(); + + if (!Directory.Exists(StoragePath)) + { + return Task.CompletedTask; + } + + foreach (var filePath in Directory.EnumerateFiles(StoragePath, $"{prefix}*.body")) + { + cancellationToken.ThrowIfCancellationRequested(); + if (Path.GetFileName(filePath).StartsWith(prefix, StringComparison.Ordinal)) + { + TryDelete(filePath); + } + } + + return Task.CompletedTask; + } + string GetBodyFilePath(string bodyId) => Path.Combine(StoragePath, $"{bodyId}.body"); static void TryDelete(string filePath) diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/FileSystemBodyStorageSettings.cs b/src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStorageSettings.cs similarity index 83% rename from src/ServiceControl.Persistence.EFCore/Abstractions/FileSystemBodyStorageSettings.cs rename to src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStorageSettings.cs index 8da22700a7..0e03a8abbf 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/FileSystemBodyStorageSettings.cs +++ b/src/ServiceControl.Persistence.BodyStorage/FileSystemBodyStorageSettings.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Abstractions; +namespace ServiceControl.Persistence.BodyStorage; public sealed class FileSystemBodyStorageSettings : BodyStorageSettings { diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/IBodyStoragePersistence.cs b/src/ServiceControl.Persistence.BodyStorage/IBodyStoragePersistence.cs similarity index 57% rename from src/ServiceControl.Persistence.EFCore/Infrastructure/IBodyStoragePersistence.cs rename to src/ServiceControl.Persistence.BodyStorage/IBodyStoragePersistence.cs index 685efe7a01..51369c6b45 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/IBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.BodyStorage/IBodyStoragePersistence.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Infrastructure; +namespace ServiceControl.Persistence.BodyStorage; using System; using System.Threading; @@ -8,8 +8,7 @@ namespace ServiceControl.Persistence.EFCore.Infrastructure; /// External storage for message bodies too large or too binary to sit inline in the database. /// /// -/// Bodies are immutable and addressed by bodyId (the UniqueMessageId) alone, so re-failures of the -/// same message resolve to the same stored body and writes can be skipped when it already exists. +/// Bodies are immutable and addressed by the caller's bodyId, so repeated writes keep the first body. /// public interface IBodyStoragePersistence { @@ -21,4 +20,11 @@ public interface IBodyStoragePersistence /// gone: callers delete the body before the row that names it. /// Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default); + + /// + /// Deletes every body whose bodyId starts with . Implementations throw + /// when a delete fails, so callers never drop the rows that name a body still in the store. + /// + /// The prefix is empty or contains a wildcard or path character. + Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default); } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/IDriveSpaceProvider.cs b/src/ServiceControl.Persistence.BodyStorage/IDriveSpaceProvider.cs similarity index 72% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/IDriveSpaceProvider.cs rename to src/ServiceControl.Persistence.BodyStorage/IDriveSpaceProvider.cs index 7dd2ffff24..a0f53a950c 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/IDriveSpaceProvider.cs +++ b/src/ServiceControl.Persistence.BodyStorage/IDriveSpaceProvider.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; public interface IDriveSpaceProvider { diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/MessageBodyFileResult.cs b/src/ServiceControl.Persistence.BodyStorage/MessageBodyFileResult.cs similarity index 77% rename from src/ServiceControl.Persistence.EFCore/Infrastructure/MessageBodyFileResult.cs rename to src/ServiceControl.Persistence.BodyStorage/MessageBodyFileResult.cs index 81ebf87a23..d8159a7515 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/MessageBodyFileResult.cs +++ b/src/ServiceControl.Persistence.BodyStorage/MessageBodyFileResult.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Infrastructure; +namespace ServiceControl.Persistence.BodyStorage; using System.IO; diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/OwnedStream.cs b/src/ServiceControl.Persistence.BodyStorage/OwnedStream.cs similarity index 97% rename from src/ServiceControl.Persistence.EFCore/Infrastructure/OwnedStream.cs rename to src/ServiceControl.Persistence.BodyStorage/OwnedStream.cs index 77b3f0693a..73c4d13b77 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/OwnedStream.cs +++ b/src/ServiceControl.Persistence.BodyStorage/OwnedStream.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Infrastructure; +namespace ServiceControl.Persistence.BodyStorage; /// /// Couples a returned stream to another resource that must remain alive while the stream is read. diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStoragePersistence.cs b/src/ServiceControl.Persistence.BodyStorage/S3BodyStoragePersistence.cs similarity index 75% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStoragePersistence.cs rename to src/ServiceControl.Persistence.BodyStorage/S3BodyStoragePersistence.cs index f5194dab85..30c4db8012 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.BodyStorage/S3BodyStoragePersistence.cs @@ -1,12 +1,10 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; using System.IO.Compression; using System.Net; using Amazon.S3; using Amazon.S3.Model; using ServiceControl.Infrastructure; -using ServiceControl.Persistence.EFCore.Abstractions; -using ServiceControl.Persistence.EFCore.Infrastructure; public class S3BodyStoragePersistence : IBodyStoragePersistence { @@ -96,6 +94,35 @@ public async Task WriteBody(string bodyId, ReadOnlyMemory body, string con public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default) => client.DeleteObjectAsync(bucketName, Key(bodyId), cancellationToken); + public async Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + BodyIdPrefix.Validate(prefix); + + var request = new ListObjectsV2Request { BucketName = bucketName, Prefix = Key(prefix) }; + ListObjectsV2Response page; + do + { + page = await client.ListObjectsV2Async(request, cancellationToken); + + if (page.S3Objects is { Count: > 0 }) + { + var deleted = await client.DeleteObjectsAsync(new DeleteObjectsRequest + { + BucketName = bucketName, + Objects = page.S3Objects.Select(o => new KeyVersion { Key = o.Key }).ToList() + }, cancellationToken); + + if (deleted.DeleteErrors is { Count: > 0 }) + { + var failed = deleted.DeleteErrors[0]; + throw new InvalidOperationException($"Failed to delete {deleted.DeleteErrors.Count} object(s) with prefix '{request.Prefix}', first '{failed.Key}': {failed.Code} {failed.Message}"); + } + } + + request.ContinuationToken = page.NextContinuationToken; + } while (page.IsTruncated == true); + } + async Task Exists(string key, CancellationToken cancellationToken) { try diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/S3BodyStorageSettings.cs b/src/ServiceControl.Persistence.BodyStorage/S3BodyStorageSettings.cs similarity index 91% rename from src/ServiceControl.Persistence.EFCore/Abstractions/S3BodyStorageSettings.cs rename to src/ServiceControl.Persistence.BodyStorage/S3BodyStorageSettings.cs index 97af894a07..43969ef3ad 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/S3BodyStorageSettings.cs +++ b/src/ServiceControl.Persistence.BodyStorage/S3BodyStorageSettings.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.EFCore.Abstractions; +namespace ServiceControl.Persistence.BodyStorage; public sealed class S3BodyStorageSettings : BodyStorageSettings { diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3ClientFactory.cs b/src/ServiceControl.Persistence.BodyStorage/S3ClientFactory.cs similarity index 90% rename from src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3ClientFactory.cs rename to src/ServiceControl.Persistence.BodyStorage/S3ClientFactory.cs index 42aa2ae45c..441a4bb8a8 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3ClientFactory.cs +++ b/src/ServiceControl.Persistence.BodyStorage/S3ClientFactory.cs @@ -1,9 +1,8 @@ -namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +namespace ServiceControl.Persistence.BodyStorage; using Amazon; using Amazon.Runtime; using Amazon.S3; -using ServiceControl.Persistence.EFCore.Abstractions; static class S3ClientFactory { diff --git a/src/ServiceControl.Persistence.BodyStorage/ServiceControl.Persistence.BodyStorage.csproj b/src/ServiceControl.Persistence.BodyStorage/ServiceControl.Persistence.BodyStorage.csproj new file mode 100644 index 0000000000..b641089344 --- /dev/null +++ b/src/ServiceControl.Persistence.BodyStorage/ServiceControl.Persistence.BodyStorage.csproj @@ -0,0 +1,23 @@ + + + + net10.0 + enable + enable + true + true + + + + + + + + + + + + + + + diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistenceConfiguration.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistenceConfiguration.cs index 2418cef18e..826534b47b 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistenceConfiguration.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistenceConfiguration.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql; +using ServiceControl.Persistence.BodyStorage; using Microsoft.Extensions.Logging; using ServiceControl.Infrastructure; using ServiceControl.Persistence.EFCore.Abstractions; diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj b/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj index 1f72d54fe7..378ac07b0b 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj @@ -28,6 +28,7 @@ + diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj b/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj index 7706488219..807aff19fe 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj @@ -28,6 +28,7 @@ + diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistenceConfiguration.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistenceConfiguration.cs index 741888b1c8..10e5310b80 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistenceConfiguration.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistenceConfiguration.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.SqlServer; +using ServiceControl.Persistence.BodyStorage; using Microsoft.Extensions.Logging; using ServiceControl.Infrastructure; using ServiceControl.Persistence.EFCore.Abstractions; diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs index 4b76e5add7..4206a418b9 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.BodyStorage; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; using NServiceBus.Unicast.Subscriptions.MessageDrivenSubscriptions; diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs index 5bf874154a..fe7b1b36a5 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.BodyStorage; using Microsoft.Extensions.Logging; using ServiceControl.Configuration; using ServiceControl.Infrastructure; diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs index 1d57e8de46..19353d167f 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs @@ -1,5 +1,7 @@ namespace ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.BodyStorage; + public abstract class EFPersisterSettings : PersistenceSettings { public static readonly TimeSpan MigrationCommandTimeout = TimeSpan.FromMinutes(40); diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStorageInstaller.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStorageInstaller.cs index d0aa969eb7..575a094a07 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStorageInstaller.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStorageInstaller.cs @@ -1,10 +1,10 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; using ServiceControl.Persistence; -using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.BodyStorage; public class AzureBlobBodyStorageInstaller(AzureBlobBodyStorageSettings settings) : IBodyStorageInstaller { public Task Provision(CancellationToken cancellationToken = default) => - AzureBlobClientFactory.CreateContainerClient(settings).CreateIfNotExistsAsync(cancellationToken: cancellationToken); + BodyStorageProvisioner.Provision(settings, cancellationToken); } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs index 01b40b4b8a..36f72f488e 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; +using ServiceControl.Persistence.BodyStorage; using ServiceControl.Infrastructure; using System.Linq.Expressions; using System.Text; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs index 2766e08e3d..d76a6cbdcd 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs @@ -1,14 +1,10 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; using ServiceControl.Persistence; -using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.BodyStorage; public class FileSystemBodyStorageInstaller(FileSystemBodyStorageSettings settings) : IBodyStorageInstaller { - public Task Provision(CancellationToken cancellationToken = default) - { - Directory.CreateDirectory(settings.StoragePath); - - return Task.CompletedTask; - } + public Task Provision(CancellationToken cancellationToken = default) => + BodyStorageProvisioner.Provision(settings, cancellationToken); } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStorageInstaller.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStorageInstaller.cs index 767fbda461..3f7c4f84f9 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStorageInstaller.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStorageInstaller.cs @@ -1,19 +1,10 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; -using Amazon.S3.Model; -using Amazon.S3.Util; using ServiceControl.Persistence; -using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.BodyStorage; public class S3BodyStorageInstaller(S3BodyStorageSettings settings) : IBodyStorageInstaller { - public async Task Provision(CancellationToken cancellationToken = default) - { - using var client = S3ClientFactory.Create(settings); - - if (!await AmazonS3Util.DoesS3BucketExistV2Async(client, settings.BucketName)) - { - await client.PutBucketAsync(new PutBucketRequest { BucketName = settings.BucketName }, cancellationToken); - } - } + public Task Provision(CancellationToken cancellationToken = default) => + BodyStorageProvisioner.Provision(settings, cancellationToken); } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/EFEnvironmentDataProvider.cs b/src/ServiceControl.Persistence.EFCore/Implementation/EFEnvironmentDataProvider.cs index 0b3809c362..663c6dcc34 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/EFEnvironmentDataProvider.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/EFEnvironmentDataProvider.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Implementation; +using ServiceControl.Persistence.BodyStorage; using System; using System.Globalization; using Abstractions; @@ -111,9 +112,9 @@ static string Seconds(TimeSpan value) => static string BodyStorageType(BodyStorageSettings bodyStorage) => bodyStorage switch { - FileSystemBodyStorageSettings => nameof(Abstractions.BodyStorageType.FileSystem), - AzureBlobBodyStorageSettings => nameof(Abstractions.BodyStorageType.AzureBlob), - S3BodyStorageSettings => nameof(Abstractions.BodyStorageType.S3), + FileSystemBodyStorageSettings => nameof(ServiceControl.Persistence.BodyStorage.BodyStorageType.FileSystem), + AzureBlobBodyStorageSettings => nameof(ServiceControl.Persistence.BodyStorage.BodyStorageType.AzureBlob), + S3BodyStorageSettings => nameof(ServiceControl.Persistence.BodyStorage.BodyStorageType.S3), _ => "Unknown" }; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/FailedErrorImportDataStore.cs b/src/ServiceControl.Persistence.EFCore/Implementation/FailedErrorImportDataStore.cs index 547e5e4678..c6ecbe2104 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/FailedErrorImportDataStore.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/FailedErrorImportDataStore.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Implementation; +using ServiceControl.Persistence.BodyStorage; using System.IO; using System.Text.Json; using Microsoft.EntityFrameworkCore; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs index 186c1e8790..5c1d77a3d3 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; +using ServiceControl.Persistence.BodyStorage; using System.Collections.Concurrent; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs index bd4d4a2d9d..249c254713 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; +using ServiceControl.Persistence.BodyStorage; using Microsoft.Extensions.DependencyInjection; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFRecoverabilityIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFRecoverabilityIngestionUnitOfWork.cs index 2b9f37cab5..532cf81ecc 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFRecoverabilityIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFRecoverabilityIngestionUnitOfWork.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; +using ServiceControl.Persistence.BodyStorage; using EntityConfigurations; using NServiceBus; using NServiceBus.Transport; diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs index ad5082de5c..6ac0231a55 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.BodyStorage; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; diff --git a/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj b/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj index 9ecbf964d0..17be246222 100644 --- a/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj +++ b/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj @@ -16,12 +16,10 @@ + - - - diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/PersistenceTestsContext.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/PersistenceTestsContext.cs index 4ce91496d5..b4a0b5c1c7 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/PersistenceTestsContext.cs +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/PersistenceTestsContext.cs @@ -2,6 +2,7 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Linq; diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/SchemaMustExistTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/SchemaMustExistTests.cs index 26f51e2a5b..a561e2530c 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/SchemaMustExistTests.cs +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/SchemaMustExistTests.cs @@ -1,6 +1,7 @@ // ReSharper disable once CheckNamespace namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Threading.Tasks; diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/PersistenceTestsContext.cs b/src/ServiceControl.Persistence.Tests.SqlServer/PersistenceTestsContext.cs index 9ded3182bc..968c8a20d9 100644 --- a/src/ServiceControl.Persistence.Tests.SqlServer/PersistenceTestsContext.cs +++ b/src/ServiceControl.Persistence.Tests.SqlServer/PersistenceTestsContext.cs @@ -1,6 +1,7 @@ // ReSharper disable once CheckNamespace namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Linq; diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/SchemaMustExistTests.cs b/src/ServiceControl.Persistence.Tests.SqlServer/SchemaMustExistTests.cs index 32a09c87e4..86f18e4ba3 100644 --- a/src/ServiceControl.Persistence.Tests.SqlServer/SchemaMustExistTests.cs +++ b/src/ServiceControl.Persistence.Tests.SqlServer/SchemaMustExistTests.cs @@ -1,6 +1,7 @@ // ReSharper disable once CheckNamespace namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Threading.Tasks; diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/SqlServerDatabaseHostingProbeTests.cs b/src/ServiceControl.Persistence.Tests.SqlServer/SqlServerDatabaseHostingProbeTests.cs index 11780a2066..a2f0aef91e 100644 --- a/src/ServiceControl.Persistence.Tests.SqlServer/SqlServerDatabaseHostingProbeTests.cs +++ b/src/ServiceControl.Persistence.Tests.SqlServer/SqlServerDatabaseHostingProbeTests.cs @@ -5,6 +5,7 @@ namespace ServiceControl.Persistence.Tests.SqlServer; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging.Abstractions; using NUnit.Framework; +using ServiceControl.Persistence.BodyStorage; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.SqlServer; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/AzureBlobBodyStorageTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/AzureBlobBodyStorageTests.cs index 6f8ec73091..41271396bc 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/AzureBlobBodyStorageTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/AzureBlobBodyStorageTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Text; @@ -113,6 +114,33 @@ public async Task Delete_removes_the_body() Assert.That(await store.ReadBody(bodyId), Is.Null); } + [Test] + public async Task Delete_with_prefix_removes_only_matching_bodies() + { + var store = await CreateContainer(); + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("2026102010-a", body, "text/plain"); + await store.WriteBody("2026102010-b", body, "text/plain"); + await store.WriteBody("2026102011-a", body, "text/plain"); + + await store.DeleteBodiesWithPrefix("2026102010"); + + Assert.Multiple(async () => + { + Assert.That(await store.ReadBody("2026102010-a"), Is.Null); + Assert.That(await store.ReadBody("2026102010-b"), Is.Null); + Assert.That(await store.ReadBody("2026102011-a"), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_with_prefix_rejects_a_blank_prefix() + { + var store = await CreateContainer(); + + Assert.ThrowsAsync(() => store.DeleteBodiesWithPrefix("")); + } + [Test] public async Task Delete_of_a_missing_body_does_not_throw() { diff --git a/src/ServiceControl.Persistence.Tests/EFCore/BodyStorageConfigurationTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/BodyStorageConfigurationTests.cs index a9f37514d3..a87ff8f673 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/BodyStorageConfigurationTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/BodyStorageConfigurationTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.Collections.Generic; using NUnit.Framework; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs index 070056f6d1..d22b464b82 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Text; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataConfigurationTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataConfigurationTests.cs index 12bda43295..6d9febca46 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataConfigurationTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataConfigurationTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.Collections.Generic; using System.Threading; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataTests.cs index c899ad2df3..68d0c2b9f1 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/EnvironmentDataTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.Collections.Generic; using System.Linq; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/ErrorIngestionTestBase.cs b/src/ServiceControl.Persistence.Tests/EFCore/ErrorIngestionTestBase.cs index fd84861ed2..aeca08e36a 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/ErrorIngestionTestBase.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/ErrorIngestionTestBase.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.Collections.Generic; using System.Linq; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/EventsRetentionConfigurationTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/EventsRetentionConfigurationTests.cs index 35f1d7220b..eac054642c 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/EventsRetentionConfigurationTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/EventsRetentionConfigurationTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using NUnit.Framework; using ServiceControl.Configuration; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/ExpectedLengthStreamTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/ExpectedLengthStreamTests.cs index 67570b2add..05aee45060 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/ExpectedLengthStreamTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/ExpectedLengthStreamTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System.IO; using NUnit.Framework; using ServiceControl.Persistence.EFCore.Infrastructure; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/FileSystemBodyStorageCustomCheckTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/FileSystemBodyStorageCustomCheckTests.cs index 835dace309..5e9816743f 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/FileSystemBodyStorageCustomCheckTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/FileSystemBodyStorageCustomCheckTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Linq; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs b/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs index e03d5c130b..14ae447e25 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs @@ -1,6 +1,8 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; +using System.Linq; using System.Collections.Generic; using System.IO; using System.Threading; @@ -101,5 +103,21 @@ public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToke return Task.CompletedTask; } + public Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(prefix); + + lock (gate) + { + foreach (var bodyId in store.Keys.Where(k => k.StartsWith(prefix, StringComparison.Ordinal)).ToList()) + { + deleted.Add(bodyId); + store.Remove(bodyId); + } + } + + return Task.CompletedTask; + } + public record StoredBody(string BodyId, byte[] Body, string ContentType); } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/QueryTimeoutConfigurationTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/QueryTimeoutConfigurationTests.cs index e9f6697274..ddc8ef2424 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/QueryTimeoutConfigurationTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/QueryTimeoutConfigurationTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using NUnit.Framework; using ServiceControl.Configuration; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/S3BodyStorageTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/S3BodyStorageTests.cs index 1dedb815e1..c221b9c639 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/S3BodyStorageTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/S3BodyStorageTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using ServiceControl.Persistence.BodyStorage; using System; using System.IO; using System.Text; @@ -110,6 +111,33 @@ public async Task Delete_removes_the_body() Assert.That(await store.ReadBody(bodyId), Is.Null); } + [Test] + public async Task Delete_with_prefix_removes_only_matching_bodies() + { + var store = await CreateBucket(); + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("2026102010-a", body, "text/plain"); + await store.WriteBody("2026102010-b", body, "text/plain"); + await store.WriteBody("2026102011-a", body, "text/plain"); + + await store.DeleteBodiesWithPrefix("2026102010"); + + Assert.Multiple(async () => + { + Assert.That(await store.ReadBody("2026102010-a"), Is.Null); + Assert.That(await store.ReadBody("2026102010-b"), Is.Null); + Assert.That(await store.ReadBody("2026102011-a"), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_with_prefix_rejects_a_blank_prefix() + { + var store = await CreateBucket(); + + Assert.ThrowsAsync(() => store.DeleteBodiesWithPrefix("")); + } + [Test] public async Task Delete_of_a_missing_body_does_not_throw() { diff --git a/src/ServiceControl.slnx b/src/ServiceControl.slnx index e9a507b944..87721d0685 100644 --- a/src/ServiceControl.slnx +++ b/src/ServiceControl.slnx @@ -45,6 +45,7 @@ + diff --git a/tools/testing-tool/README.md b/tools/testing-tool/README.md index 2ecacf82f7..dab6c5a6f9 100644 --- a/tools/testing-tool/README.md +++ b/tools/testing-tool/README.md @@ -12,7 +12,7 @@ The tool runs an NServiceBus endpoint (`TestingTool.Load`) that sends messages through a handler which fails based on the active scenario. Failed messages are routed to the `error` queue for ServiceControl to ingest. Each scenario throws a real exception type from its own method, so ServiceControl's default grouping (exception type + first stack frame) -puts each failure shape in its own group, and immediate retries are disabled so groups stay clean. Five scenarios are built in: +puts each failure shape in its own group, and immediate retries are disabled so groups stay clean. Six scenarios are built in: | Scenario | Category | Failure shape | |---|---|---| @@ -21,6 +21,7 @@ puts each failure shape in its own group, and immediate retries are disabled so | `poison-message` | Poison | 15% deterministic always-fail messages — retry storm | | `deserialization-failure` | Deserialization | 100% fail with `SerializationException` (bad deployment) | | `background-noise` | Noise | ~3% always-on baseline — rotates through four exception types (four small groups) | +| `large-messages` | Body | Starts at 3 msg/s (rate is configurable), each body about 256 KiB of JSON text (the wire envelope is slightly larger), above the 100 KB external body storage threshold. 20% deterministic failures with `InvalidDataException`; the other 80% succeed and are audited | Scenarios are controlled from the web UI, or via: - `GET /api/scenarios` — list scenarios with live status (category, rate, error counts) @@ -111,7 +112,7 @@ tools/testing-tool/ TestingTool/ # ASP.NET Core host — API endpoints, web UI, NServiceBus endpoint, # scenario runner, bypass writer, recoverability/search jobs, # OTel setup, ServiceControl REST client - TestingTool.Scenarios/ # IScenario contract + the five scenario implementations + TestingTool.Scenarios/ # IScenario contract + the six scenario implementations TestingTool.Contracts/ # shared DTOs (scenarios, status, jobs, bypass) TestingTool.AppHost/ # Aspire AppHost — orchestrates platform, tool, and observability stack obs/ # observability config: OTel Collector, Prometheus, Grafana provisioning @@ -189,6 +190,30 @@ separator) or `--name:value` (colon separator): | `--tag` | *(none — uses the current build's image)* | any image tag, e.g. `pr-1234` or `6.3.1` | Override the ServiceControl container image tag (useful for testing PR-based prereleases) | | `--error-ingestion-scale-unit` | `0` | non-negative integer | Number of additional error-ingestion-only scale-out instances to spin up alongside the primary error instance (each runs with `--error-ingestion-only`) | | `--audit-instances` | `1` | non-negative integer | Number of ServiceControl.Audit instances to add. They take the same `--persistence` backend, report custom checks to the primary and export OTel to the collector | +| `--azureblob-body-storage` | *(none — uses file-system body storage)* | `""` | Store message bodies in Azure Blob Storage instead of the container file system. Applies only with `--persistence:SqlServer` or `--persistence:PostgreSql`. Quote the value because connection strings contain semicolons. The error instance and its scale-out units share the `error-bodies` container, and each audit instance gets its own `audit-bodies-` container | +| `--s3-body-storage` | *(none — uses file-system body storage)* | `` | Store message bodies in an S3 bucket instead of the container file system. Applies only with `--persistence:SqlServer` or `--persistence:PostgreSql`, and cannot be combined with `--azureblob-body-storage`. Setup creates the bucket if it is missing and the credentials permit it. The error instance and its scale-out units share the `error-bodies/` key prefix, and each audit instance gets its own `audit-bodies-/` prefix | +| `--s3-region` | SDK default | region name, e.g. `eu-west-1` | Region for `--s3-body-storage`. Requires `--s3-body-storage` | +| `--s3-service-url` | AWS | absolute `http` or `https` URL | S3-compatible endpoint, e.g. MinIO or LocalStack. The ServiceControl containers connect to it, so it must be reachable from inside a container. Requires `--s3-body-storage` | +| `--s3-access-key-id`, `--s3-secret-access-key` | *(none — default AWS credential chain)* | key pair | Static credentials for `--s3-body-storage`. Set both or neither. Without them, the AWS default credential chain runs inside each container; the host's credentials are not forwarded. Static keys appear in the container environment | + +Pass static credentials explicitly, or omit both key options when the containers can use the AWS default credential chain: + +```bash +aspire run tools/testing-tool/TestingTool.AppHost/TestingTool.AppHost.csproj -- \ + --persistence PostgreSql --s3-body-storage my-bodies --s3-region eu-west-1 \ + --s3-access-key-id "$AWS_ACCESS_KEY_ID" --s3-secret-access-key "$AWS_SECRET_ACCESS_KEY" +``` + +For MinIO or LocalStack, use an endpoint reachable from the containers. This example assumes `host.docker.internal` resolves to the host: + +```bash +aspire run tools/testing-tool/TestingTool.AppHost/TestingTool.AppHost.csproj -- \ + --persistence SqlServer --s3-body-storage bodies --s3-region us-east-1 \ + --s3-service-url http://host.docker.internal:9000 \ + --s3-access-key-id minioadmin --s3-secret-access-key minioadmin +``` + +The static credential options do not support session tokens. For temporary AWS credentials, configure the default credential chain inside the containers instead. The Aspire dashboard provides allocated ports for each service. The testing tool automatically connects to ServiceControl via the platform's transport and REST API URL, and sends its OTLP diff --git a/tools/testing-tool/TestingTool.AppHost/AppHost.cs b/tools/testing-tool/TestingTool.AppHost/AppHost.cs index 7c94c661b4..6f58f2fe01 100644 --- a/tools/testing-tool/TestingTool.AppHost/AppHost.cs +++ b/tools/testing-tool/TestingTool.AppHost/AppHost.cs @@ -4,6 +4,12 @@ var options = CliOptions.Parse(args); var persistenceType = options.GetValue("persistence", PersistenceType.RavenDb); var imageTag = options.GetValueOrDefault("tag"); +var azureBlobBodyStorage = options.GetValueOrDefault("azureblob-body-storage"); +if (azureBlobBodyStorage is not null && string.IsNullOrWhiteSpace(azureBlobBodyStorage)) +{ + throw new ArgumentException("--azureblob-body-storage requires a non-blank Azure Storage connection string."); +} +var s3BodyStorage = S3BodyStorageOptions.Parse(options, azureBlobBodyStorage is not null); Console.WriteLine($"Using persistence type: {persistenceType}"); var builder = DistributedApplication.CreateBuilder(args); @@ -28,7 +34,7 @@ .WithEnvironment("SERVICECONTROL_ALLOWMESSAGEEDITING", "true") .WithEnvironment("SERVICECONTROL_DISABLEEXTERNALINTEGRATIONSPUBLISHING", "true") .WithEnvironment("OTEL_EXPORTER_OTLP_ENDPOINT", observability.Collector.GetEndpoint("otlp-grpc")) - .WithPersistenceType(persistenceType) + .WithPersistenceType(persistenceType, azureBlobBodyStorage, s3BodyStorage) .WithRunMode(PlatformRunMode.SetupAndRun); for (int i = 0; i < options.GetValue("error-ingestion-scale-unit", 0); i++) { @@ -46,7 +52,7 @@ .WithEnvironment("SERVICECONTROL_ERRORINGESTIONBATCHTIMEOUT", "00:00:00.100") //.WaitFor(primaryErrorInstance) - .WithPersistenceType(persistenceType) + .WithPersistenceType(persistenceType, azureBlobBodyStorage, s3BodyStorage) .WithRunMode(PlatformRunMode.Run); } @@ -70,7 +76,7 @@ .WithAuditQueueName(i == 0 ? "audit" : "audit" + i) .WithRunMode(PlatformRunMode.SetupAndRun) .WithIngestionTuning() - .WithPersistenceType(persistenceType); + .WithPersistenceType(persistenceType, azureBlobBodyStorage, s3BodyStorage); } // --- Testing tool --- diff --git a/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs b/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs index 740db8ce37..973bc47eb8 100644 --- a/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs +++ b/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs @@ -44,7 +44,7 @@ public static IResourceBuilder AddPostgresPersist } public static IResourceBuilder WithPersistenceType( - this IResourceBuilder error, PersistenceType type) + this IResourceBuilder error, PersistenceType type, string? azureBlobConnectionString = null, S3BodyStorageOptions? s3 = null) { if (type == PersistenceType.RavenDb) { @@ -59,11 +59,21 @@ public static IResourceBuilder WithPersiste _ => throw new ArgumentOutOfRangeException(nameof(type), type, null) }; + // The primary and its scale-out units share one database and body storage location. + error = azureBlobConnectionString is not null + ? error + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "AzureBlob") + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_AZURE_CONNECTIONSTRING", azureBlobConnectionString) + : s3 is not null + ? error + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "S3") + .WithS3Environment("SERVICECONTROL_MESSAGEBODY_S3_", s3) + : error + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem") + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_FILESYSTEM_STORAGEPATH", "/tmp/ServiceControlBodyStorage"); + return error .WaitFor(db) - //file storage for now - .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem") - .WithEnvironment("SERVICECONTROL_MESSAGEBODY_FILESYSTEM_STORAGEPATH", "/tmp/ServiceControlBodyStorage") .WithEnvironment("SERVICECONTROL_PERSISTENCETYPE", PersistenceTypeName(type)) .WithEnvironment("SERVICECONTROL_DATABASE_CONNECTIONSTRING", db); } @@ -71,9 +81,7 @@ public static IResourceBuilder WithPersiste /// - /// Points a ServiceControl.Audit instance at the same SQL Server or PostgreSQL container the - /// primary uses. The keys are namespaced to the audit instance, and there is no body storage: - /// the audit EF persisters store bodies inline in the audit row, capped by MaxBodySizeToStore. + /// Points an Audit instance at the primary's database container with the configured body storage. /// /// /// The setting root namespace is "ServiceControl.Audit", and the settings reader rewrites both @@ -83,7 +91,7 @@ public static IResourceBuilder WithPersiste /// so the namespaced form wins. /// public static IResourceBuilder WithPersistenceType( - this IResourceBuilder audit, PersistenceType type) + this IResourceBuilder audit, PersistenceType type, string? azureBlobConnectionString = null, S3BodyStorageOptions? s3 = null) { if (type == PersistenceType.RavenDb) { @@ -97,6 +105,21 @@ public static IResourceBuilder WithPersiste _ => throw new ArgumentOutOfRangeException(nameof(type), type, null) }; + // Audit instances need separate body storage locations; see docs/message-body-storage.md. + audit = azureBlobConnectionString is not null + ? audit + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_STORAGETYPE", "AzureBlob") + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_AZURE_CONNECTIONSTRING", azureBlobConnectionString) + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_AZURE_CONTAINERNAME", "audit-bodies-" + audit.Resource.Name) + : s3 is not null + ? audit + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_STORAGETYPE", "S3") + .WithS3Environment("SERVICECONTROL_AUDIT_MESSAGEBODY_S3_", s3) + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_S3_KEYPREFIX", "audit-bodies-" + audit.Resource.Name + "/") + : audit + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_STORAGETYPE", "FileSystem") + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_FILESYSTEM_STORAGEPATH", "/tmp/ServiceControlAuditBodyStorage"); + return audit .WaitFor(db) .WithEnvironment("SERVICECONTROL_AUDIT_PERSISTENCETYPE", PersistenceTypeName(type)) @@ -143,6 +166,30 @@ public static IResourceBuilder AddTransport( } + static IResourceBuilder WithS3Environment(this IResourceBuilder builder, string prefix, S3BodyStorageOptions s3) + where T : IResourceWithEnvironment + { + builder = builder.WithEnvironment(prefix + "BUCKETNAME", s3.Bucket); + if (s3.Region is not null) + { + builder = builder.WithEnvironment(prefix + "REGION", s3.Region); + } + + if (s3.ServiceUrl is not null) + { + builder = builder.WithEnvironment(prefix + "SERVICEURL", s3.ServiceUrl); + } + + if (s3.AccessKeyId is not null && s3.SecretAccessKey is not null) + { + builder = builder + .WithEnvironment(prefix + "ACCESSKEYID", s3.AccessKeyId) + .WithEnvironment(prefix + "SECRETACCESSKEY", s3.SecretAccessKey); + } + + return builder; + } + static string PersistenceTypeName(PersistenceType persistence) => persistence switch { PersistenceType.SqlServer => "SQLServer", diff --git a/tools/testing-tool/TestingTool.AppHost/S3BodyStorageOptions.cs b/tools/testing-tool/TestingTool.AppHost/S3BodyStorageOptions.cs new file mode 100644 index 0000000000..1833c67d09 --- /dev/null +++ b/tools/testing-tool/TestingTool.AppHost/S3BodyStorageOptions.cs @@ -0,0 +1,54 @@ +namespace TestingTool.AppHost; + +/// +/// S3 body storage settings from the --s3-* command-line options. Without keys, the container uses the AWS default credential chain. +/// +public record S3BodyStorageOptions(string Bucket, string? Region, string? ServiceUrl, string? AccessKeyId, string? SecretAccessKey) +{ + /// null when --s3-body-storage is not set. + public static S3BodyStorageOptions? Parse(CliOptions options, bool azureBlobSelected) + { + var bucket = Read(options, "s3-body-storage"); + var region = Read(options, "s3-region"); + var serviceUrl = Read(options, "s3-service-url"); + var accessKeyId = Read(options, "s3-access-key-id"); + var secretAccessKey = Read(options, "s3-secret-access-key"); + + if (bucket is null) + { + if (region is not null || serviceUrl is not null || accessKeyId is not null || secretAccessKey is not null) + { + throw new ArgumentException("--s3-region, --s3-service-url, --s3-access-key-id, and --s3-secret-access-key require --s3-body-storage."); + } + + return null; + } + + if (azureBlobSelected) + { + throw new ArgumentException("--azureblob-body-storage and --s3-body-storage are mutually exclusive."); + } + + if ((accessKeyId is null) != (secretAccessKey is null)) + { + throw new ArgumentException("--s3-access-key-id and --s3-secret-access-key must be set together."); + } + + if (serviceUrl is not null && !(Uri.TryCreate(serviceUrl, UriKind.Absolute, out var uri) && (uri.Scheme == Uri.UriSchemeHttp || uri.Scheme == Uri.UriSchemeHttps))) + { + throw new ArgumentException("--s3-service-url must be an absolute http or https URL."); + } + + return new S3BodyStorageOptions(bucket, region, serviceUrl, accessKeyId, secretAccessKey); + } + + static string? Read(CliOptions options, string key) + { + if (!options.TryGetValue(key, out var value)) + { + return null; + } + + return string.IsNullOrWhiteSpace(value) ? throw new ArgumentException($"--{key} requires a non-blank value.") : value; + } +} diff --git a/tools/testing-tool/TestingTool.Scenarios/IScenario.cs b/tools/testing-tool/TestingTool.Scenarios/IScenario.cs index eb314ac683..3edf42c67c 100644 --- a/tools/testing-tool/TestingTool.Scenarios/IScenario.cs +++ b/tools/testing-tool/TestingTool.Scenarios/IScenario.cs @@ -21,6 +21,9 @@ public interface IScenario /// Default recommended emission rate in messages/second. double DefaultRate { get; } + /// Fixed message body size in bytes, or null for the default 3–6 KB range. + int? BodySizeBytes { get; } + /// Activity source used to emit OpenTelemetry traces for this scenario's work. ActivitySource ActivitySource { get; } diff --git a/tools/testing-tool/TestingTool.Scenarios/LargeMessagesScenario.cs b/tools/testing-tool/TestingTool.Scenarios/LargeMessagesScenario.cs new file mode 100644 index 0000000000..3e42797b5c --- /dev/null +++ b/tools/testing-tool/TestingTool.Scenarios/LargeMessagesScenario.cs @@ -0,0 +1,28 @@ +using System.Runtime.CompilerServices; + +namespace TestingTool.Scenarios; + +/// +/// Large message scenario: every body is about 256 KiB, above the default 100 KB external body storage +/// threshold. A deterministic subset of messages (by hash) fails, so both the Audit and Error ingestion +/// paths store large bodies. +/// +public sealed class LargeMessagesScenario(string shardId) : ScenarioBase(shardId) +{ + public override string Name => "large-messages"; + public override string Description => "Publishes about 256 KiB message bodies and fails 20% of them. Exercises external body storage on the Audit and Error paths."; + public override string Category => "Body"; + public override double DefaultRate => 3; + public override int? BodySizeBytes => 256 * 1024; + + private const double FailureRatio = 0.20; + + public override bool ShouldFail(string messageId) => Hash(messageId) < FailureRatio; + + public override Exception CreateException() => + Capture(ProcessLargePayload, "large:payload-too-large"); + + [MethodImpl(MethodImplOptions.NoInlining)] + private static void ProcessLargePayload() => + throw new InvalidDataException("The large message payload could not be processed."); +} diff --git a/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs b/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs index 8a9576bcee..f38bc38767 100644 --- a/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs +++ b/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs @@ -62,10 +62,16 @@ public static class MessageTextGenerator /// mixed with filler prose. The result is a JSON object with /// several string fields so it resembles a realistic integration message body, and the /// embedded terms are guaranteed to be present in . + /// When is set the body grows to at least that many UTF-8 bytes instead. /// - public static string GenerateBody(long sequence) + public static string GenerateBody(long sequence, int? targetBytes = null) { - var targetSize = Random.Shared.Next(MinSize, MaxSize + 1); + if (targetBytes is { } requested) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(requested, nameof(targetBytes)); + } + + var targetSize = targetBytes ?? Random.Shared.Next(MinSize, MaxSize + 1); // Choose a random subset of searchable terms to embed in this body. Embedding several // per message means every term is covered frequently across a run, so searches drawn diff --git a/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs b/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs index 28bd70b2f9..3b05566ce5 100644 --- a/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs +++ b/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs @@ -20,6 +20,7 @@ protected ScenarioBase(string shardId) public abstract string Description { get; } public abstract string Category { get; } public virtual double DefaultRate => 10; + public virtual int? BodySizeBytes => null; public ActivitySource ActivitySource { get; } public virtual TimeSpan? Cooldown => null; diff --git a/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs b/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs index 9a15884619..ea051f8120 100644 --- a/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs +++ b/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs @@ -197,9 +197,9 @@ private async Task WriteLoop(BypassRun run, int workerIndex, CancellationToken c { while (!ct.IsCancellationRequested && run.TryClaim(out var seq)) { - // Plausible JSON body (~3–6 KB) seeded with searchable terms so the bypass path + // Plausible JSON body (scenario-sized, default ~3–6 KB) seeded with searchable terms so the bypass path // also feeds ServiceControl's full-text search index with real content. - var textBody = MessageTextGenerator.GenerateBody(seq); + var textBody = MessageTextGenerator.GenerateBody(seq, scenario.BodySizeBytes); var message = new LoadMessage { Sequence = seq, TextBody = textBody }; diff --git a/tools/testing-tool/TestingTool/Program.cs b/tools/testing-tool/TestingTool/Program.cs index 1f93e62640..6ea727445d 100644 --- a/tools/testing-tool/TestingTool/Program.cs +++ b/tools/testing-tool/TestingTool/Program.cs @@ -31,6 +31,7 @@ builder.Services.AddSingleton(_ => new PoisonMessageScenario(shardId)); builder.Services.AddSingleton(_ => new DeserializationScenario(shardId)); builder.Services.AddSingleton(_ => new RandomBackgroundNoiseScenario(shardId)); +builder.Services.AddSingleton(_ => new LargeMessagesScenario(shardId)); builder.Services.AddSingleton(); builder.Services.AddSingleton(); diff --git a/tools/testing-tool/TestingTool/ScenarioRunner.cs b/tools/testing-tool/TestingTool/ScenarioRunner.cs index 76b6d62a34..a7879903bc 100644 --- a/tools/testing-tool/TestingTool/ScenarioRunner.cs +++ b/tools/testing-tool/TestingTool/ScenarioRunner.cs @@ -174,9 +174,8 @@ private async Task SendOneAsync(ScenarioRuntime runtime, long seq, CancellationT { var scenario = runtime.Scenario; - // Plausible JSON body (~3–6 KB) seeded with searchable terms so ServiceControl's - // full-text search index has real content to exercise. - var textBody = MessageTextGenerator.GenerateBody(seq); + // Searchable terms ensure full-text queries have indexed content to match. + var textBody = MessageTextGenerator.GenerateBody(seq, scenario.BodySizeBytes); // Deterministic message id shared with the handler: FailingMessageHandler calls // scenario.ShouldFail(context.MessageId), so setting the id here ensures the