Start with the task and the information it touches
A useful vendor review starts with a specific workflow: taking an enquiry, finding an approved document, or preparing a draft for staff. List what information enters the system, where it goes, who can access it and who checks the output. A product name or an AI label is not enough to answer those questions.
The ABA published Formal Opinion 512 on July 29, 2024. It discusses professional duties including competence, confidentiality, communication, candor, supervision and reasonable fees when lawyers use generative AI. It is general guidance, not a certification of a product. Read it alongside the rules that apply to the firm; a vendor checklist is not a legal determination.
Separate training, processing and retention
Processing means using information to produce an answer or carry out a task. Training means using data to update a model. Retention means keeping records after processing. These are different activities. A no-training commitment does not, by itself, mean prompts, responses, files or logs are immediately deleted.
Ask for the terms that cover the exact service and account configuration being proposed. Map the full route, including the model provider, integration platform, speech service, transcript storage, CRM and backups where applicable. A control on one service does not establish the behaviour of the rest.
Nine questions to put in the scope
1. What is the system allowed to do?
Identify the input, expected output and permitted actions. Write down actions requiring approval and requests the system must hand to a person. Keep those decisions specific enough to test.
2. Which records can it access?
Record each connection, account and access role. Test that the proposed user can reach the intended material and cannot retrieve another matter’s restricted records.
3. How is information used and retained?
Separate training use, operational logging, storage, deletion and backup retention. Request the applicable documentation and settings, including exceptions. Avoid assuming that a dedicated endpoint guarantees a particular retention policy.
4. Which providers receive information?
List the services involved, their purpose and the records sent to them. Have the firm’s responsible reviewer decide which contracts and data protections are required for that use.
5. What security evidence covers this system?
Request evidence appropriate to the proposed access and data: relevant audit reports, access controls, incident handling and testing records. Check dates, scope and exclusions. A badge does not show that your particular workflow is correctly configured.
6. How will answers be checked?
Use representative tasks with expected results. Check whether cited sources support the answer, whether uncertainty is visible and whether an incorrect answer reaches a person before an action. Record failures as well as successful demonstrations.
7. What happens when the connection fails?
Test a timeout, unavailable record and rejected write. Agree where unresolved work appears, who receives it and how a retry avoids duplicate messages or records.
8. Who operates and changes it?
Name the owner of permissions, provider subscriptions, updates and incident response. Agree what the build includes and which support work requires a separate scope.
9. What can the firm take with it?
Specify the agreed workflows, documentation, configuration, export formats and operating instructions. Identify custom deliverables and third-party licences separately. Test the export and handover rather than relying on an ownership slogan.
An intake pilot needs its own acceptance test
For a fictional intake pilot, start with approved administrative questions and a clear human handover. Use test enquiries with incomplete details, urgent wording, requests for legal advice and an unavailable staff member. Inspect the summary against the original test conversation, then verify that the task reaches the right person. This example is a technical test plan, not a claim about a Benian law-firm deployment.
Record the baseline before changing a process. Count reviewed enquiries, correct handovers, unresolved items and staff effort. Do not promise a multiple of qualifying information or assume that a transcript proves an enquiry was handled correctly.
Keep the legal and billing decisions with the firm
Opinion 512 discusses review of AI output, risks to client information, disclosure and fees. Its fee discussion distinguishes tool costs that function as overhead from some matter-specific expenses. The firm should assess its agreements and applicable obligations rather than automatically passing through a subscription or billing time it did not spend. Consult the opinion’s full text for context.
Bring evidence to the scope discussion
Use the AI Project Checklist and Excel workbook to record access, tests, owners and open questions. The agent permissions guide adds practical checks for tool access and untrusted input. These resources help prepare a technical review; they do not certify compliance.
Benian’s AI Consulting service can help scope a proposed business workflow. For a law-firm use, the responsible lawyer needs to review the professional obligations and approve the intended operation. A scoping call starts with the task, existing tools and constraints; it does not establish legal suitability.
Reviewed September 8, 2026. Legal context is linked to the ABA’s Formal Opinion 512, dated July 29, 2024. The nine technical questions and fictional test plan are Benian’s implementation guidance, not ABA requirements, certification criteria or a reported client result. Check current local professional rules before using confidential client information.