Skip to main content

Test your app

Tests that keep the guarantees of the policy and the row rules - no raw functions, no leaks between tenants, a budget for each call - and tests of scheduled work and agents.

The policy and the row rules protect every function that goes through defineFunctions. Three tests keep it that way while the application grows: no raw functions, no leaks between tenants, and a budget for each call. The last two sections test scheduled work and agents. All tests run in convex-test with the real Better Auth component, so nothing is mocked except the deployment and outside services.

The first sections show the tests of starters/mcp-oauth-agent: the short files in full, and the shape of the long ones with a link to the file. The tests of the last two sections use the app of Start here and Internal operations.

Set up convex-test

bash
pnpm add -D convex-test vitest
vitest.config.ts
import { defineConfig } from 'vitest/config'

export default defineConfig({
  test: { environment: 'node', include: ['convex/**/*.test.ts'] },
})
convex/test.setup.ts
/// <reference types="vite/client" />

import betterAuth from '@lupinum/better-convex-nuxt/better-auth/test'
import { convexTest } from 'convex-test'
import { vi } from 'vitest'

import schema from './schema'

export const modules = import.meta.glob('./**/*.ts', { eager: false })

/** convex-test with the real Better Auth component. */
export function initConvexTest() {
  // The issuer and the resource of an MCP grant (`grantMcp`) come from these.
  vi.stubEnv('SITE_URL', 'https://app.example.test')
  vi.stubEnv('CONVEX_SITE_URL', 'https://deployment.example.test')
  const t = convexTest(schema, modules)
  betterAuth.register(t)
  return t
}

Convex does not deploy files with two dots in the name, such as test.setup.ts and leaks.test.ts.

The tests import ./_generated/api. After you add a module or an export, run pnpm exec better-convex-nuxt convex codegen. It needs a deployment; a local one works without an account or network. See Code generation without a cloud deployment.

signInAs(t, 'ann') from @lupinum/better-convex-nuxt/better-auth/test returns a client that calls functions as that person. It creates two rows in the Better Auth component, or reuses them when they exist:

RowIDOther fields
userannname ann, email ann@example.com
sessionann-sessionexpires in one hour (expiresInMs option)

Insert your app's own users row yourself, with authId: 'ann'. When the session was deleted, for example to test a sign-out, the next signInAs creates a session with a new ID.

The session expires one hour after signInAs, also in fake time. With vi.useFakeTimers(), t.finishAllScheduledFunctions(vi.runAllTimers) runs every timer, so it also moves the clock past the session's end, and the next call is signed out. Move the clock in small steps instead, or give the session a longer life. Pass expiresInMs on the first signInAs for that person in the test: a later call reuses the live session and keeps its expiry.

ts
// Run the scheduled work of the next 10 seconds, one second at a time. Work
// that a step schedules starts in a later step, so take a few steps more than
// the time of the last job.
for (let step = 0; step < 12; step += 1) {
  vi.advanceTimersByTime(1_000)
  await t.finishInProgressScheduledFunctions()
}

// Or: a session that outlives a job scheduled 24 hours later.
const ann = await signInAs(t, 'ann', { expiresInMs: 7 * 24 * 60 * 60 * 1000 })

The loop does not work for a scheduled action that awaits a timer itself, such as setTimeout: the test waits for the action while the clock stands still. Use t.finishAllScheduledFunctions(vi.runAllTimers) with a longer session there.

Assert codes, not messages

A refused call rejects with a ConvexError whose data.code says why: NOT_FOUND, FORBIDDEN, APPROVAL_NOT_FOUND. Compare the code with one literal value. The codes are part of the public API. The message texts are not, and a release can change them.

No raw functions

A function built with Convex's own query, mutation, internalMutation or httpAction gets no actor, no policy and no row rules. An operation that calls a raw internal function fails, but a raw public function is open to every caller. This test finds each one.

convex/no-bypass.test.ts
import { unguardedFunctions } from '@lupinum/better-convex-functions/test'
import { expect, test } from 'vitest'

import { modules } from './test.setup'

// Catches a function built with Convex's own builders instead of the ones from ./functions:
// it would skip the caller check, the policy and the row rules. Mark a deliberate one with
// `trusted(reason, fn)`.
test('every function goes through the policy and the row rules', async () => {
  const trustedRoutes = { '/api/auth/': 'Better Auth endpoints; the auth library checks each one.' }
  expect(await unguardedFunctions(modules, { trustedRoutes })).toEqual([])
})

unguardedFunctions takes the module map that convexTest receives, so it checks the modules that Convex deploys. Like Convex, it skips files with two dots in the name, _generated/ at the root, and a local component (a folder with its own convex.config.ts). It throws when the map is empty or has no defineFunctions operation, because then the test would check nothing.

When a function must skip the library, for example an operator function or a trigger of the auth component, wrap it and say why: trusted('Operator function: run it from the dashboard.', internalMutation({ … })). The reason shows up when someone reviews the list of trusted functions. convex/connections.ts and convex/auth.ts in the starter show both kinds.

trustedRoutes does the same for HTTP routes under a path prefix that another library checks, such as the Better Auth endpoints. A prefix covers whole path segments: /api/auth covers /api/auth/session, not /api/authz/users.

Run this test in CI. It is the guarantee: the runtime checks are the second line. See Threat model.

Launch check

launchProblems runs the checks that matter most before an app goes live. It returns one plain sentence for each problem, and an empty list when all is well.

convex/launch.test.ts
import { launchProblems } from '@lupinum/better-convex-functions/test'
import { expect, test } from 'vitest'

import crons from './crons'
import { fns } from './functions'
import schema from './schema'
import { modules } from './test.setup'

// Catches the mistakes that hurt after launch: a raw Convex function, a table that keeps a
// person's data after they delete their account, housekeeping nobody schedules, a public write
// without a limit, and a tool that no agent sees. Each problem names the fix.
test('the app has no launch problems', async () => {
  const trustedRoutes = { '/api/auth/': 'Better Auth endpoints; the auth library checks each one.' }
  expect(await launchProblems({ modules, schema, crons, fns, trustedRoutes })).toEqual([])
})

It checks five things, and nothing else:

  1. Raw functions. The same check as unguardedFunctions. trustedRoutes is passed on.
  2. Account deletion. Every field that holds v.id('users') is covered by an entry in the erasure map of defineFunctions (or the table has keep), and a module exports fns.erasure.eraseStep. The check also finds an ID inside an object, an array or a record (as key or value). Erasure cannot reach such an ID, so only keep covers it, or you store the ID in its own indexed field. The check looks for v.id('users'), so the table of your users must be called users. That table and the library tables need no entry. Without erasure, the sentence says that account deletion is not set up.
  3. Housekeeping. If the agents package built a housekeeping function in your modules, a cron in crons calls it.
  4. Limits on public writes. Every public action that a public mutation uses has a limits entry in the policy.
  5. Tools in a catalog. Every operation with a tool is in a module that you pass to defineTools, and the module that calls defineTools exports its result as tools. Otherwise no agent sees the tool.

Each sentence names the file, table or action and the fix. To accept a finding on purpose, use the way that exists already: trusted(reason, fn), erasure: { table: { keep: reason } }, or a limit.

It does not check that your rules, roles or limits are the right ones, that beforeDelete refuses what it should, or anything outside Convex, such as files in storage or other services.

No leaks between tenants

Put one row of another tenant in the database: the canary. Then let people who must not reach it aim every call at it. Each row of the table names an actor, a call and the one code the call must fail with:

ActorCallsCode
A stranger: no membership in the tenantevery callNOT_FOUND
A former member: membership status removedevery callNOT_FOUND
A lower role: a viewer in the tenantevery writeFORBIDDEN
Each of them, through their own agentcheck_approval with the ID of another agent's requestAPPROVAL_NOT_FOUND

A stranger and a former member get NOT_FOUND, not FORBIDDEN: the answer must not tell them that the row exists.

The starter's convex/leaks.test.ts has one call for each operation, through the web client, and one for each tool, through callTool (see Tools and approvals). Each call takes the IDs it aims at:

ts
const operations: Record<Exclude<keyof typeof projects, 'organizations'>, Call> = {
  search: (_t, eve, { organizationId }) =>
    eve.web.query(api.projects.search, { organizationId, paginationOpts }),
  rename: (_t, eve, { projectId }) =>
    eve.web.mutation(api.projects.rename, { projectId, name: 'Mine' }),
  // … one entry for each operation
}
const toolCalls: Record<string, Call> = {
  archive_project: (t, eve, { projectId }) =>
    callTool(t, tools, eve.principal, 'archive_project', { projectId }),
  // … one entry for each tool that takes an ID
}

const rows: [Actor, string, string][] = [
  ['stranger', 'rename', 'NOT_FOUND'],
  ['former member', 'archive_project', 'NOT_FOUND'],
  ['viewer', 'rename', 'FORBIDDEN'],
  // … one row for each actor and call
]

Each row runs three checks:

ts
it.each(rows)('a %s: %s fails with %s', async (actor, name, expected) => {
  const { t, bob, eve } = await setup(actor)
  const call = calls[name]!
  const before = await state()

  await expect(call(t, eve, bob).then(() => 'no error', code)).resolves.toBe(expected)
  // Nothing changed: no project renamed, archived or added, and no new request for a person.
  await expect(state()).resolves.toEqual(before)

  // The positive control: the same call through the same path succeeds with Eve's own IDs.
  await expect(call(t, eve, eve)).resolves.toBeDefined()
})
  • One literal code. A check such as "the result does not name the canary" passes for any refusal, also for INVALID_INPUT from a wrong argument name. Then the call never reached the tenant check.
  • The positive control. The same call with the actor's own IDs must succeed. It proves that the call is correct and that only the foreign ID makes it fail.
  • Nothing changed. The canary is unchanged, no row was added to the other tenant, and no approval request waits there.

Test the tools separately. A tool runs the same operation, but an agent's call takes another path: a write that needs approval becomes a request for a person, and a request for another tenant's row must fail before it reaches that tenant's approval queue.

Record<Exclude<keyof typeof projects, 'organizations'>, …> makes the test fail to compile when a new operation has no call. organizations takes no ID; the same file checks that it lists only the organizations of active memberships. For the tools, one test compares the call names with tools.catalog.

keyof typeof names every export of the module, also internal operations, which a person cannot call. Keep internal operations in a module of their own, as convex/summaryWork.ts in Internal operations. When a module must hold both, leave the internal ones out by name. For a sites module with the internal operations runCheck and saveResult: Record<Exclude<keyof typeof sites, 'runCheck' | 'saveResult'>, …>.

When an operation takes two IDs, add a row that mixes them: one of the actor's own tenant and one of the canary's.

A budget for each call

countDocuments from @lupinum/better-convex-functions/test counts the documents one call reads and writes, with everything the library does around the handler: the session, the user, the role and the tenant rows. The starter's convex/cost.test.ts sets up one organization with three projects and counts two calls:

ts
// Catches a change that makes a call read or write more than it did: an extra lookup per row,
// a missing index, a write in a read path. Each number is the budget; raise it on purpose.
test('a page of three projects', async () => {
  const { ann, organizationId } = await setup()
  const cost = await countDocuments(() =>
    ann.query(api.projects.search, {
      organizationId,
      paginationOpts: { numItems: 10, cursor: null },
    }),
  )
  // The session check (2), user, organization, membership, then the three rows of the page.
  expect(cost).toEqual({ reads: 8, writes: 0 })
})

Each number is a budget. A change that reads more, for example a lookup per row instead of one per call, fails the test and must raise the number on purpose. Cost lists what each kind of call costs.

countDocuments follows convex-test: a query counts the rows it returns, not the rows that a .filter() skipped. A deployment counts those too. Do not run calls in parallel while you count.

The counts include the library's own reads, so they can change when you upgrade Better Convex. The changelog says so under "Tests". Check the new numbers and update the literals.

Scheduled functions

convex-test runs scheduled functions with setTimeout. Fake only setTimeout and clearTimeout, and keep the real clock: with a fake clock, running the timers moves the time past the expiry of the session and of an agent's grant, and the scheduled call fails with NOT_SIGNED_IN or AGENT_DISABLED.

convex/summaries.test.ts
import { signInAs } from '@lupinum/better-convex-nuxt/better-auth/test'
import { afterEach, beforeEach, expect, test, vi } from 'vitest'

import { api } from './_generated/api'
import { initConvexTest } from './test.setup'

beforeEach(() => {
  vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] })
  // The summary service is outside the app: answer for it.
  vi.stubGlobal(
    'fetch',
    vi.fn(async () => Response.json({ summary: 'Plans for the next year.' })),
  )
})
afterEach(() => {
  vi.useRealTimers()
  vi.unstubAllGlobals()
})

/** Runs scheduled work, and the work it schedules, until none is left. */
async function runScheduled(t: ReturnType<typeof initConvexTest>) {
  for (let step = 0; step < 10; step++) {
    if (vi.getTimerCount() === 0) return
    vi.runOnlyPendingTimers()
    await t.finishInProgressScheduledFunctions()
  }
  throw new Error('Scheduled work still runs after 10 steps.')
}

// Catches a broken chain from the request to the saved summary: the scheduled action, its read
// and its write, each as the person who asked.
test('a requested summary is saved on the project', async () => {
  const t = initConvexTest()
  const projectId = await t.run(async (ctx) => {
    const userId = await ctx.db.insert('users', { authId: 'ann', name: 'Ann' })
    const organizationId = await ctx.db.insert('organizations', { name: 'Acme' })
    await ctx.db.insert('memberships', { organizationId, userId, role: 'owner' })
    return await ctx.db.insert('projects', { organizationId, name: 'Roadmap', status: 'active' })
  })
  const ann = await signInAs(t, 'ann')
  await ann.mutation(api.summaries.request, { projectId })

  await runScheduled(t)

  await expect(t.run((ctx) => ctx.db.get(projectId))).resolves.toMatchObject({
    summary: 'Plans for the next year.',
  })
})

runScheduled starts what is scheduled, waits until it is done, and repeats for the work that schedules more work. The step limit stops a chain that never ends.

An agent's grant from grantMcp lasts 10 minutes, like an access token. For scheduled work that runs longer, give the session and the grant a longer life:

ts
await signInAs(t, 'ann', { expiresInMs: 4 * 60 * 60 * 1000 }) // the session grantMcp reuses
const principal = {
  ...(await grantMcp(t, 'ann', ['projects:read'])),
  expiresAt: Math.floor(Date.now() / 1000) + 4 * 60 * 60,
}

Tools and approvals

grantMcp(t, 'ann', scopes) from @lupinum/better-convex-nuxt/better-auth/test gives Ann a live MCP grant: a host client test-host, the resource ${CONVEX_SITE_URL}/mcp, and her consent to the scopes. It returns the principal that the door passes to a tool after it checks an access token. The grant uses the same user and session as signInAs, in either order, so deleting the session ends both, as a sign-out does.

callTool(t, tools, principal, name, input) from @lupinum/better-convex-agents/test calls a tool as the MCP door does: only a tool that the principal's scopes unlock, with request_id taken from input. The input goes through JSON, as a host's does: NaN arrives as null. It resolves with the tool's output: { status: 'done', result }, or { status: 'needs_approval', approvalId, summary, url }. It rejects with the tool's own error, as t.mutation does. A tool that the scopes do not unlock throws an Error that lists the tools they unlock.

Three tests cover what only agents have.

The agent surface. One literal of every tool, its scopes and its approval rule. A scope that unlocks a write tool by accident, or an approval rule that was dropped, changes the literal. Review a change to it like a change to the policy. The test below is for the app of Start here.

An agent's request. It waits for a person, and the approval runs it.

convex/tools.test.ts
import { callTool } from '@lupinum/better-convex-agents/test'
import { grantMcp, signInAs } from '@lupinum/better-convex-nuxt/better-auth/test'
import { expect, test } from 'vitest'

import { api } from './_generated/api'
import { tools } from './agents'
import { initConvexTest } from './test.setup'

// Catches a scope that unlocks a write tool by accident, and an approval rule that was dropped.
test('gives agents these tools, with these scopes and approvals', () => {
  expect(
    tools.catalog.map(({ name, kind, scopes, approval }) => ({ name, kind, scopes, approval })),
  ).toEqual([
    { name: 'list_projects', kind: 'query', scopes: ['projects:read'], approval: 'never' },
    { name: 'archive_project', kind: 'mutation', scopes: ['projects:write'], approval: 'always' },
    { name: 'check_approval', kind: 'query', scopes: [], approval: 'never' },
  ])
})

// Catches an agent that archives without a person, and an approval that does not run the archive.
test('an agent’s archive waits for a person, then runs as the agent', async () => {
  const t = initConvexTest()
  const projectId = await t.run(async (ctx) => {
    const userId = await ctx.db.insert('users', { authId: 'ann', name: 'Ann' })
    const organizationId = await ctx.db.insert('organizations', { name: 'Acme' })
    await ctx.db.insert('memberships', { organizationId, userId, role: 'owner' })
    return await ctx.db.insert('projects', { organizationId, name: 'Roadmap', status: 'active' })
  })
  const principal = await grantMcp(t, 'ann', ['projects:read', 'projects:write'])
  const ann = await signInAs(t, 'ann')

  const asked = await callTool(t, tools, principal, 'archive_project', { projectId })
  if (asked.status !== 'needs_approval') throw new Error('The archive did not wait for a person.')
  expect(asked.summary).toBe('Archive the project "Roadmap".')
  await expect(t.run((ctx) => ctx.db.get(projectId))).resolves.toMatchObject({ status: 'active' })

  await expect(ann.mutation(api.agents.approve, { approvalId: asked.approvalId })).resolves.toEqual(
    { status: 'approved' },
  )
  await expect(t.run((ctx) => ctx.db.get(projectId))).resolves.toMatchObject({
    status: 'archived',
  })
})

Who may approve. A table of people who try to approve another person's agent request, with the result for each. The starter's convex/agents.test.ts lets owners and admins decide (approvers: { 'projects.archive': ['owner', 'admin'] }):

ts
it.each([
  ['an owner', 'approved', 'owner', 'active'],
  ['an admin', 'approved', 'admin', 'active'],
  ['a member', 'APPROVAL_NOT_FOUND', 'member', 'active'],
  ['a viewer', 'APPROVAL_NOT_FOUND', 'viewer', 'active'],
  ['a former admin', 'APPROVAL_NOT_FOUND', 'admin', 'removed'],
  ['a stranger', 'APPROVAL_NOT_FOUND', null, null],
] as const)('%s who approves gets %s', async (_name, expected, role, status) => {
  // … Ann's agent asks to archive; Pat, with this role, tries to approve it.
  expect(outcome).toBe(expected)
  // A refused approval leaves the project as it was.
  await expect(t.run((ctx) => ctx.db.get(projectId))).resolves.toMatchObject({
    organizationId,
    status: expected === 'approved' ? 'archived' : 'active',
  })
})

A principal with fewer scopes stands for an access token with fewer scopes: grantMcp(t, 'ann', ['projects:read']). The starter's convex/authorization.test.ts uses grantMcp and callTool to check that each tool call stops after the app suspends the person or removes the membership.