Start with rule-based automation when the inputs are clear and the decision is fixed. Consider an AI pilot when the task involves varied language or cases that simple rules do not handle well, provided suitable data, approved access and a workable review process exist. If the underlying process is unclear or nobody can review the output, address that first.
Readiness is not established by buying a subscription or collecting a large folder of documents. The practical question is whether a specific task can be tested against the current approach and whether the evidence can support a decision about using it.
When is simple automation the better starting point?
Sending a reminder after a recorded date, routing a request by the branch a customer selected or transferring a field between systems can often begin with explicit rules. Interpreting an ambiguous free-text message or preparing a summary for review may justify testing a language model.
Google’s engineering guidance recommends considering simpler approaches rather than introducing machine learning for its own sake, and establishing useful measurement early. That comparison principle does not imply every AI project needs a custom-trained model. Google’s Rules of Machine Learning.
| Task | Suggested starting point | What to test |
|---|---|---|
| Reminder for a known appointment | Scheduling and rules | Correct timing and duplicate prevention |
| Routing by a selected branch | Explicit conditions and integration | Complete fields and correct permissions |
| Classifying differently worded complaints | Compare rules with an AI candidate | Errors by category and ambiguous cases |
| Summarizing a long conversation | A human-reviewed draft | Missing facts and unsupported additions |
| Making a consequential account change | An approved workflow with controls | Authorization, verification, recovery and action records |
A combined approach may work: a model proposes a category, rules select the destination, and a person approves sensitive cases. The model does not need to run the whole process.
What should you check before authorizing a pilot?
1. A bounded task and a baseline
Describe the input, output and user: “Suggest a category for a completed conversation so a customer-service supervisor can review it.” That is more testable than “improve company efficiency with AI.”
Measure the current approach, including understanding and correction time and the work requiring intervention. Without a baseline, apparent improvement is difficult to interpret. Check whether the actual bottleneck is missing information or delayed human approval rather than the speed of generating text.
2. Suitable data that can be used for the task
Check where the files came from, when they were updated, who owns them and who authorizes their use. Data being inside the company does not establish that every employee or supplier can use it for any purpose.
For an Arabic-language task, include samples that reflect actual users: formal Arabic, relevant colloquial wording, spelling errors, short messages and mixed Arabic–English text where it occurs. Avoid selecting only clean examples that make a demonstration impressive. Keep development material separate from the evaluation set, define expected outcomes and allow an “unclear” category when appropriate.
3. Access, integration and action limits
Will the system only read information, or can it write changes? Can it access one team’s files or the entire organization? What happens if a connected system becomes unavailable?
API Integration & Development needs defined access boundaries and failure handling even if the model performs well in a chat window. Testing the model alone does not establish that the connected workflow is safe or reliable.
4. An operational owner and a realistic review process
Identify who resolves disputed cases, receives alerts and can disable the pilot. Ask the people who will use the system whether the proposed review workload fits their actual responsibilities.
Address data, security and contractual requirements for the specific case with the appropriate specialists before real use. This guide does not prescribe a mandatory hosting location or provide compliance certification. NIST describes its AI Risk Management Framework as voluntary; using it does not constitute accreditation. NIST framework overview.
Worked hypothetical example: Arabic customer-conversation classification
This is an illustrative scenario, not a reported LAB project result.
A Saudi business wants to understand why customers contact it before building automated replies. A sensible starting point is classification of historical conversations approved for this use, without sending messages to customers or changing their orders.
Initial categories: pre-purchase enquiry, order follow-up, complaint and human review required. But a message such as “يعطيكم العافية، طلبت من أسبوع وما وصل” combines a polite opening with a delivery problem: the customer ordered a week ago and has not received it. One conversation may include both a follow-up and a complaint. The team must decide whether multiple labels are allowed or one primary reason should be selected.
Review disputed examples before evaluating the model. If reviewers disagree without a resolution rule, the task definition also needs work. That disagreement is not sufficient evidence of model failure on its own.
The comparison: the team labels a separate evaluation set, then compares a simple rules approach and a candidate model on those same cases. Review missed complaints, unnecessary escalations of ordinary enquiries and human correction time. Do not replace these details with an attractive overall percentage.
A possible hypothetical decision: the model helps with straightforward enquiries but misses indirectly worded complaints. Keep the work internal, improve category definitions and example coverage, then evaluate again. A convincing demonstration is not a reason to authorize automatic customer replies.
Write an acceptance sheet before building
| Item | What to agree before the pilot |
|---|---|
| Scope | One task, a specified channel and permitted data types |
| Comparison | The current method or a simple rules baseline on the same cases |
| Quality | Error definitions, relevant category-level measures and appropriate acceptance limits |
| Review | Who corrects outputs, when a person takes over and how outcomes are recorded |
| Cost | Model usage, integration, review and correction, not just the API request |
| Stop conditions | Events requiring immediate suspension and who decides |
| Handover | Results, errors, versions, limitations and a recommendation for the next step |
No single accuracy threshold suits every business. Choose limits according to the consequences of errors and record the sample size, composition and limits of inference. NIST’s measurement guidance suggests acceptable performance boundaries, documented tests and evaluation under conditions resembling intended use. NIST Measure, sections 1.1, 2.1 and 2.3.
Software Quality Assurance can cover integration journeys, permissions and failure states. Specialized output evaluation should be defined within the scope of AI & Machine Learning Engineering. Depending on the solution, both are needed; one does not replace the other.
What should block a move from pilot to operation?
- The data source or permission for its intended use is unresolved.
- Results look good only after difficult cases are excluded.
- Human correction consumes the expected benefit.
- There is no clear response to an outage or consequential error.
- You cannot identify the version behind an output or re-examine the case.
- Nobody owns the review of deterioration after changes.
If the pilot meets its conditions, expand in a bounded way with monitoring. Observability & Reliability Engineering can connect service health to alerts, operational records and incident procedures. Output quality also needs ongoing attention; a healthy server does not establish that the answers remain useful.
Common mistakes that make a pilot look better than it is
Testing only the demonstration examples. Include inconvenient, incomplete and ambiguous cases that the live process actually encounters.
Counting model speed but not review effort. A fast draft can still be expensive if a person must repeatedly reconstruct and correct it.
Allowing actions because classification worked. Suggesting a label and changing a customer record have different consequences and require separate decisions.
Treating a readiness score as permission to proceed. A favorable overall assessment does not resolve a missing data-use authorization or an untested failure path.
Frequently asked questions
Do we need thousands of files to start an assessment?
Not necessarily. First you need suitable examples to understand the task and its constraints. Data needs differ between using an existing model, retrieving information from documents and training a model. File count alone does not answer the question.
Is high accuracy enough to approve a pilot?
No. Examine the distribution of errors, excluded cases, review cost and permission boundaries. A strong overall result can hide weakness in a rare but important category.
Can the outcome be that we do not need AI?
Yes. A better input form, a system integration or an explicit rule may be sufficient. A useful assessment allows that conclusion rather than treating development as the only acceptable outcome.
Should we begin with automatic customer replies?
Not simply because a prototype writes convincingly. Begin with a test that does not affect customers or with employee-reviewed drafts. Consider wider authority only after the appropriate quality and controls have been demonstrated for the use case.
Your next step
Bring one task description, its data map, the current process and safe examples of difficult cases. Start with LAB’s AI Readiness Assessment to determine whether the next step is simple rules, foundational data work or a bounded pilot with explicit acceptance conditions.
Sources
- Google for Developers: Rules of Machine Learning: simpler approaches and measurement before complexity.
- NIST: AI Risk Management Framework: the framework’s voluntary status.
- NIST AI RMF Playbook: Measure: performance limits, evaluation records and test context.
- Sources reviewed on 26 October 2026. The example and tables are proposed applications, not regulatory approval or a readiness claim about any particular organization.
