Skip to main content

Testing Strategy

Testing is a first-class citizen in Ormed. We recommend a multi-layered testing strategy that includes unit tests, integration tests, and property-based tests.

The Test Harness​

To make testing easier, we use a "Harness" pattern powered by server_testing and server_testing_shelf.

The harness allows us to run tests in two modes:

  1. In-Memory: Fast, isolated tests that run without opening a real network port.
  2. Live Server: Tests that run against a real HTTP server, useful for verifying network-level behavior.

The harness sets up a fresh in-memory database, initializes the storage service, and bootstraps the Shelf server for each test. This ensures that tests are isolated and don't interfere with each other.

OrmedTestConfig createMovieCatalogConfig() {
  final baseDataSource = createDataSource();

  final config = setUpOrmed(
    dataSource: baseDataSource,
    migrationDescriptors: buildMigrations(),
    seeders: [AppDatabaseSeeder.new],
    adapterFactory: (_) => SqliteDriverAdapter.inMemory(),
    strategy: DatabaseIsolationStrategy.migrateWithTransactions,
  );

  tearDownAll(() async {
    await config.manager.cleanup();
  });

  return config;
}

class MovieCatalogTestHarness {
  MovieCatalogTestHarness({
    required this.dataSource,
    required this.requestHandler,
    required this.storage,
    required this.templates,
    required this.uploadsDir,
  });

  final DataSource dataSource;
  final ShelfRequestHandler requestHandler;
  final StorageService storage;
  final TemplateRenderer templates;
  final Directory uploadsDir;

  Future<void> dispose() async {
    await uploadsDir.delete(recursive: true);
  }
}

Future<MovieCatalogTestHarness> createMovieCatalogHarness(
  DataSource dataSource,
) async {
  final uploadsDir = await Directory.systemTemp.createTemp('movie_uploads_');
  final storage = StorageService(uploadsRoot: uploadsDir.path);
  await storage.init();

  final templates = TemplateRenderer(
    FileSystemRoot('templates', throwOnMissing: true),
    sharedData: {'app_name': 'Movie Catalog', 'year': 2025},
  );

  final logger = buildLogger();
  final httpLogger = buildHttpLogger(logger);
  final app = MovieCatalogApp(
    database: AppDatabase.fromDataSource(dataSource),
    storage: storage,
    templates: templates,
    logger: logger,
  );

  final handler = Pipeline()
      .addMiddleware(requestIdMiddleware())
      .addMiddleware(httpLogger.middleware)
      .addHandler(app.buildHandler());

  return MovieCatalogTestHarness(
    dataSource: dataSource,
    requestHandler: ShelfRequestHandler(handler),
    storage: storage,
    templates: templates,
    uploadsDir: uploadsDir,
  );
}

The ormedGroup Test Helper​

All our tests use ormedGroup as the top-level test group instead of the standard group() from the test package. This is a special test helper provided by Ormed that:

  1. Manages Database Isolation: Each ormedGroup gets its own isolated database state. Changes in one group don't affect others.
  2. Provides a DataSource: The callback receives a DataSource (ds) that's already connected and ready to use.
  3. Handles Cleanup: Automatically cleans up database resources after the test group completes.
  4. Supports Transactions: Uses transactions to roll back changes after each test, keeping tests fast and isolated.

Inside an ormedGroup, you can still use the standard group() and test() functions as normal:

ormedGroup('My tests', (ds) {
  // 'ds' is an isolated DataSource for this test group
  late MyTestHarness harness;

  setUpAll(() async {
    harness = await createHarness(ds);
  });
  
  // Standard group() works inside ormedGroup
  group('Movie API', () {
    test('creates a movie', () async {
      // ...
    });
    
    test('lists movies', () async {
      // ...
    });
  });
});

This pattern ensures that even when running tests in parallel, each ormedGroup has its own database state.

Web route tests​

Web route tests verify that our HTML pages are rendered correctly and that form submissions work as expected. We use server_testing to make HTTP requests against our server and verify the output.

Verifies that the home page lists movies.

    test('GET / renders the catalog', () async {
      await withClient(harness, (client) async {
        final response = await client.get('/');

        response
          ..assertStatus(HttpStatus.ok)
          ..assertBodyContains('Movie Catalog')
          ..assertBodyContains('City of Amber')
          ..assertBodyContains('Science Fiction');
      });
    });

API tests​

API tests focus on the JSON endpoints. Instead of using assertion libraries directly, we interact with the TestResponse object returned by all TestClient methods (getJson, postJson, etc.).

The TestResponse Object​

The TestResponse class is a powerful wrapper that provides a fluent API for inspecting and asserting on HTTP responses. It includes built-in methods for common tasks:

  • Status Assertions: assertStatus(200), assertSuccess(), assertClientError(), assertServerError().
  • Header Assertions: assertHasHeader('X-Request-ID'), assertHeader('Content-Type', 'application/json').
  • Body Assertions: assertBodyContains('Movie created'), assertBodyIsEmpty().
  • JSON Path Assertions: assertJsonPath('movie.title', 'Inception') for quick, targeted checks.

Integrating assertable_json​

For complex JSON structures, TestResponse provides the assertJson() method. This method is the bridge to the assertable_json package.

When you call assertJson(), it provides an AssertableJson callback. This allows you to use the full power of assertable_json (like scope, each, and etc()) directly on the response:

  • Strict by Default: The library expects you to account for every property in the object. This prevents "leaky" APIs where unexpected data is sent to the client.
  • The etc() Method: Use .etc() when you only care about a subset of properties. It tells the library: "I've verified the fields I care about; ignore any other properties that might exist."

Notice how we chain assertions on the response and then use assertJson for deep structural verification.

    test('GET /api/movies returns JSON list', () async {
      await withClient(harness, (client) async {
        final response = await client.get('/api/movies');

        response
          ..assertStatus(HttpStatus.ok)
          ..assertJson((json) {
            json
              ..has('movies')
              ..countBetween('movies', 1, 10)
              ..scope('movies', (movies) {
                movies.each((movie) {
                  movie
                    ..hasAll([
                      'id',
                      'title',
                      'release_year',
                      'summary',
                      'poster_url',
                      'genre',
                    ])
                    ..whereType<int>('id')
                    ..whereType<int>('release_year')
                    ..when(movie.get('genre') != null, (movie) {
                      movie.scope('genre', (genre) {
                        genre
                          ..hasAll(['id', 'name'])
                          ..whereType<int>('id')
                          ..whereType<String>('name')
                          ..etc();
                      });
                    })
                    ..etc();
                });
              })
              ..etc();
          });
      });
    });

    test('GET /api/genres returns JSON list', () async {
      await withClient(harness, (client) async {
        final response = await client.get('/api/genres');

        response
          ..assertStatus(HttpStatus.ok)
          ..assertJson((json) {
            json
              ..has('genres')
              ..countBetween('genres', 1, 10)
              ..scope('genres', (genres) {
                genres.each((genre) {
                  genre
                    ..hasAll(['id', 'name', 'description'])
                    ..whereType<String>('id')
                    ..whereType<String>('name')
                    ..etc();
                });
              })
              ..etc();
          });
      });
    });

Property tests (Stress Testing)​

Property-based testing (using property_testing) allows us to "stress test" our application by running hundreds of tests with randomly generated data.

Generators: Gen vs Chaos​

To generate test data, we use two types of generators:

  • Gen: Generates "well-behaved" data within specific bounds (e.g., a string between 1 and 40 characters, or a year between 1888 and 2030). This tests that your app handles valid input correctly.
  • Chaos: Generates "malicious" or "garbage" data (e.g., extremely long strings, negative numbers, or special characters). This is exceptionally powerful for finding edge cases in validation logic and ensuring your server never returns a 500 Internal Server Error.

How it works: Building Complex Payloads​

Property testing is most effective when you generate complex, nested data structures. We use a functional approach to build these:

  • Gen.oneOfGen: This is the secret to effective stress testing. It allows you to mix "happy path" data (Gen) with "chaotic" data (Chaos). For example, a title generator might return a valid string 90% of the time and a chaotic string 10% of the time.
  • Composition (flatMap & map): You don't just generate single values; you compose them. By using flatMap, you can chain multiple generators together to build a complete JSON payload. This ensures that every field in your request is being randomized simultaneously.

  1. Define Generators: Create individual generators for each field and compose them into a single payload generator.

          final titleGen = Gen.oneOfGen([
            Gen.string(minLength: 1, maxLength: 40),
            Chaos.string(minLength: 0, maxLength: 8),
          ]);
          final yearGen = Gen.oneOfGen([
            Gen.integer(min: 1888, max: 2030),
            Chaos.integer(min: 0, max: 3000),
          ]);
          final summaryGen = Gen.oneOfGen<String?>([
            Gen.constant(null),
            Gen.string(maxLength: 120),
          ]);
          final genreIdGen = Gen.oneOfGen<int?>([
            Gen.constant(null),
            Gen.integer(min: 1, max: 10),
            Chaos.integer(min: -50, max: 50),
          ]);

          final payloadGen = titleGen.flatMap(
            (title) => yearGen.flatMap(
              (year) => summaryGen.flatMap(
                (summary) => genreIdGen.map((genreId) {
                  return {
                    'title': title,
                    'releaseYear': year,
                    'summary': summary,
                    'genreId': genreId,
                  };
                }),
              ),
            ),
          );

  2. Define the Runner: The PropertyTestRunner takes your generator and a test function. It executes your test logic (e.g., making an API request) hundreds of times, each time with a different payload.

          final runner = PropertyTestRunner<Map<String, dynamic>>(payloadGen, (
            payload,
          ) async {
            final response = await client.postJson('/api/movies', payload);
            expect(response.statusCode, anyOf([201, 400]));
            expect(response.statusCode, lessThan(500));

            if (response.statusCode == 201) {
              response.assertJson((json) {
                json.has('movie', (movie) {
                  movie
                    ..whereType<int>('id')
                    ..whereType<String>('title')
                    ..whereType<int>('release_year')
                    ..isGreaterOrEqual('release_year', 1888)
                    ..etc();
                });
              });
            } else if (response.statusCode == 400) {
              response.assertJson((json) {
                json
                  ..hasAny(['error', 'errors'])
                  ..etc();
              });
            }
          }, PropertyConfig(numTests: 250, maxShrinks: 50));

  3. Shrinking: If a failure is found, the library automatically tries to "shrink" the failing input to the smallest possible value that still causes the failure, making it much easier to debug.

Tests how the application handles random, potentially invalid IDs in the URL.

    test('property: API survives chaotic ids', () async {
      final client = TestClient.inMemory(harness.requestHandler);

      final runner = PropertyTestRunner<String>(
        Chaos.string(minLength: 1, maxLength: 64),
        (value) async {
          final response = await client.get('/api/movies/$value');
          expect(response.statusCode, lessThan(500));
        },
      );

      final report = await runner.run();
      await client.close();
      expect(report.success, isTrue, reason: report.report);
    });

    test('property: genre routes never 500', () async {
      final client = TestClient.inMemory(harness.requestHandler);

      final runner = PropertyTestRunner<String>(
        Chaos.string(minLength: 1, maxLength: 64),
        (value) async {
          final response = await client.get('/api/genres/$value');
          expect(response.statusCode, lessThan(500));
        },
      );

      final report = await runner.run();
      await client.close();
      expect(report.success, isTrue, reason: report.report);
    });

Model Factories​

To avoid repetitive test data setup, Ormed generates Model Factories (e.g., GenreModelFactory, MovieModelFactory). Factories provide a clean way to create test instances with sensible defaults:

  • Unique Data: Factories can generate unique values for each test run.
  • Overrides: You can override specific fields while keeping defaults for others.
  • Relationships: Factories can automatically create related models.

This keeps test data explicit and reduces boilerplate.

If you want more control, you can register external factory classes that define defaults and named states in a dedicated file. This keeps test data logic organized as factories grow.

Playwright UI tests​

import { expect, test } from '@playwright/test';
// @ts-ignore
async function createMovie(page, title: string) {
  await page.goto('/movies/new');
  await page.fill('input[name="title"]', title);
  await page.fill('input[name="releaseYear"]', '2024');
  await page.selectOption('select[name="genreId"]', { label: 'Drama' });
  await page.fill(
    'textarea[name="summary"]',
    'A lighthouse keeper rebuilds a lost archive.',
  );
  await page.click('button[type="submit"]');
  await expect(page.getByRole('heading', { name: title })).toBeVisible();
}
// @ts-ignore
test('catalog page renders', async ({ page }) => {
  await page.goto('/');
  await expect(page.getByRole('heading', { name: 'Movie Catalog' })).toBeVisible();
  await expect(page.getByText('City of Amber')).toBeVisible();
});
// @ts-ignore
test('can add a movie from the form', async ({ page }) => {
  await createMovie(page, 'Shoreline Echoes');
});

// @ts-ignore
test('genre detail shows catalog entries', async ({ page }) => {
  await page.goto('/genres/1');
  await expect(page.getByRole('heading', { name: 'Drama' })).toBeVisible();
  await expect(page.getByText('Ashes in Winter')).toBeVisible();
});

// @ts-ignore
test('can edit a movie summary', async ({ page }) => {
  const title = `Edit Trail ${Date.now()}`;
  await createMovie(page, title);

  await page.click('a:has-text("Edit")');
  await page.fill('textarea[name="summary"]', 'A revised logline for the archive.');
  await page.click('button[type="submit"]');

  await expect(page.getByText('A revised logline for the archive.')).toBeVisible();
});

// @ts-ignore
test('can delete a movie from the UI', async ({ page }) => {
  const title = `Delete Trail ${Date.now()}`;
  await createMovie(page, title);

  await page.click('a:has-text("Delete")');
  await expect(page.getByRole('heading', { name: `Delete ${title}?` })).toBeVisible();
  await page.click('button:has-text("Yes, delete")');

  await expect(page.getByRole('heading', { name: 'Movie Catalog' })).toBeVisible();
  await expect(page.getByText(title)).toHaveCount(0);
});

Run from example/fullstack/playwright:

npm install
npx playwright install
npm test