Consider a proposed assistant for Ali Reza Rashidi’s professional website. A visitor asks, “Who is Ali Reza Rashidi, and where can I find his website?” A useful answer gives the approved professional introduction and the correct address. Another visitor asks for a birthday that the approved profile does not contain. A useful answer acknowledges that missing information. Both behaviors belong in the goal. A fluent biography that adds an unsupported degree should fail even if its opening sentence is correct.

For this running example, the public website identifies Ali Reza Rashidi as a data scientist, AI architect, MLOps professional, and author. Its address is alirezarashidi.com. Those details were checked for this article. The assistant, dataset, release thresholds, and experiments described here are proposed designs; no model has been trained or benchmarked for this article.[1]

These thresholds are example product decisions, not Google requirements or measured achievements. Define the denominator for every rate. “95% instruction compliance” needs a known set of requests and an agreed rubric. Keep critical failures separate from the average: many correct introductions cannot compensate for publishing an invented home address. Conversely, an assistant that refuses every question should fail the known-fact slice, even if it never fabricates anything.

Store a small approved profile as the source of truth, with a revision date and an owner. Fine-tuning can teach a response pattern, but an updated website address should not require retraining the entire assistant. Supplying current facts at runtime makes that update explicit and reviewable. This is an architectural choice for this example: train the habit of using evidence and handling gaps, while keeping changeable facts in a maintained record.

Start with the model you could realistically deploy, using the approved profile and a clear instruction. Add a few demonstrations covering an ordinary introduction, a website-only request, and an unknown personal detail. Google’s model-adaptation guidance places prompt engineering and in-context learning before more specialized tuning, and its supervised-tuning documentation describes cases where prompting already produces the desired result. Training is a candidate when a meaningful failure remains.[2][3]

Use only the approved profile supplied with this request.
Introduce Ali Reza Rashidi accurately and concisely.
For his website, return https://alirezarashidi.com/.
If a requested detail is absent, say it is not in the profile.
Do not infer a degree, employer, birthday, or home address.
Answer in the user's requested language.

Try the prompt on development questions, then examine the failures by type. If the URL is wrong because the supplied profile is stale, fix the profile. If the correct record was never retrieved, inspect retrieval. If the evidence is present but the model repeatedly invents a credential or ignores the requested language, curated supervised examples may address a behavioral problem. This diagnosis keeps a data-access failure from being mistaken for a need to modify weights.

Keep both an initial prompt and the strongest prompt you settle on, with their outputs. The latter is the important competitor for fine-tuning. You do not establish a useful gain by comparing a carefully trained assistant against an untuned model that never received the same facts. Once the prompt is selected, freeze it for the controlled comparison. If you later optimize the prompt separately for each model, report that as an additional comparison between complete systems.

Write examples that resemble requests a real visitor would make. Include short and long questions, spelling variations, mixed requests, and explicit attempts to make the assistant guess. Correct answers should show how to use available facts and how to respond when a requested detail is absent. Vary the language naturally; copying one introduction hundreds of times creates volume without much additional behavioral coverage.

Split the examples into Train for fitting parameters, Validation for selecting prompts or training settings, and Test for the final assessment. Google’s machine-learning guidance explains why a separate validation set is needed when repeated tuning would otherwise adapt your choices to the test set. Make the partition before augmentation: paraphrases of the same seed question should remain together.[4]

Group by the source question, conversation, or generation seed, then review exact and near duplicates across groups. Normalized text hashes catch literal repeats; similarity searches can help find close paraphrases, but still need judgment. “Where is his website?” and “What is his site address?” may be too similar to support a strong generalization claim when separated only by random row assignment. Keep all turns of a conversation in one partition.

The approved profile can legitimately appear in every partition because production requests also receive that context. The prohibited overlap is an evaluation answer disguised as a near-identical training example. Do not remove the essential task context merely to make every string unique. Instead, hold out meaningful combinations: a new phrasing, a mixed known-and-unknown request, or a language condition represented sufficiently to evaluate it.

Choose split sizes from coverage needs and the number of independent groups available. There is no percentage that rescues an unrepresentative dataset. A small demonstration set can test the mechanics, but a final test with two unknown-fact questions says little about the breadth of uncertainty handling. Save stable example IDs, group IDs, split assignments, profile revision, and review notes. These make a suspicious improvement traceable.

Run the selected untuned model and prompt on the development evaluation before training. Save the complete input, response, model revision, decoding settings, and per-case scores. Google’s adaptation lab explicitly includes evaluating the base model before tuning so improvements can be quantified. A vague recollection that the original answers were poor is not a baseline.[2]

Use separate checks for factual support, instruction compliance, website correctness, and unsupported personal claims. An exact URL check is appropriate when the output is supposed to contain only a URL, provided you define acceptable normalization such as a trailing slash. A professional introduction needs a claim-level rubric: are its statements supported, and does it omit something the request explicitly required? Word overlap alone cannot establish those properties.

Have a reviewer inspect a representative sample and all critical failures. If you use an LLM judge, give it the same approved evidence and a concrete rubric, retain its reasoning, and compare its decisions with human judgments. Treat disagreement as a signal to investigate. A judge can reward a polished answer containing an invented qualification unless factual support is an explicit criterion.

Keep Test sealed during this development baseline. You may generate and store baseline test answers without viewing them, or run both fixed candidates when final evaluation begins. What matters is that neither those answers nor their scores influence prompt selection, example writing, or checkpoint choice. Report per-slice counts as well as rates so a weak unknown-fact category cannot disappear inside a large set of easy introductions.

Measure time to first token, output generation rate, total response time, and peak GPU memory. Google’s accelerator benchmarking guidance includes TTFT, inter-token latency, and tokens per second, while stressing that performance depends on the complete workload and platform. For this assistant, the user-facing question is how long a visitor waits for a useful introduction or a clear acknowledgment of missing information.[5]

Write down the timing convention. In a streaming client, TTFT can mean elapsed time from sending a request to receiving its first output token. That includes effects such as network travel and queuing. For N output tokens, a post-first-token rate can be defined as (N − 1) divided by the time between the first and final tokens. N divided by total request time is a different rate. Neither should be silently presented as aggregate server throughput.

Warm up the model until initialization and compilation effects are understood, then repeat the workload. Report cold-start behavior separately if your deployment can scale to zero. Keep the hardware, precision, inference engine, prompt lengths, output limits, batching, concurrency, and cache policy fixed for the controlled comparison. Record actual input and output token counts: a fine-tuned model that answers in fewer words may finish earlier without decoding each token faster.

Retain individual observations and summarize their distribution, including p50 and p95 with the number of requests. A short local run can reveal a large slowdown, but gives an unstable estimate of rare delays. For client timing, use actual streaming events and document whether the stream exposes tokens or buffered text chunks. For device timing, CUDA work is asynchronous; synchronize at the chosen measurement boundaries or use appropriate CUDA events.[6]

Reset peak-memory statistics after warm-up and before the measured phase, while documenting what is already resident. PyTorch’s max_memory_allocated reports peak memory occupied by tensors, not total GPU process usage; reserved allocator memory and device-level reporting answer different questions. Label each measurement precisely, and benchmark inference separately from training. Measuring a forward pass while optimizer states remain resident would misrepresent the serving footprint.[7]

A conversational record becomes a token sequence through the model’s chat template. Role delimiters, end markers, and generation prefixes are part of that contract. Use the template for the selected model family, inspect its rendered text, and preserve the same conversation structure at inference. Hugging Face warns that adding special tokens again after formatting a chat template can duplicate them and hurt performance.[8]

Here the intended objective is answer-only supervision. The profile and question remain visible as context, but their own target positions are excluded from the loss. In the usual Hugging Face causal-language-model setup, −100 is the ignored label value. This is different from the attention mask: hiding the profile from attention would prevent the assistant from using the evidence it is supposed to summarize. The loss mask says which predictions teach; the attention mask governs usable context.[14]

Inspect input_ids, attention_mask, and labels from the actual collated batch, including a padded example and one near the maximum sequence length. Decode the tokens whose labels are active, retaining special tokens during this inspection. You should recognize the intended response and the appropriate boundary tokens. Confirm that every training example has some supervised response tokens; truncation can otherwise leave a long profile and question with nothing useful to learn from.

Do not infer correctness from a trainer flag alone. Current TRL documentation distinguishes conversational assistant_only_loss from completion-only training for prompt-completion datasets. Assistant-only masking requires a compatible template that marks assistant generation spans, with automatic support limited to specified model families. Check your installed version and template. For multi-turn records, decide whether every assistant turn or only the final answer is supervised.[9]

If you construct masks yourself, compare token sequences rather than rendered character lengths, and verify the boundary on complete examples. Separately tokenized strings are not automatically a reliable prefix of the full formatted conversation. Also confirm whether the model implementation shifts causal labels internally; shifting labels twice trains the wrong prediction. Start with unpacked examples, verify boundaries and stopping behavior, then recheck the actual batch if you enable packing.

Before a long run, perform a small forward and backward pass, inspect finite loss and gradients, and verify that the expected trainable parameters receive updates. A tiny overfit exercise can help diagnose a broken pipeline, but its memorized answers are not evidence of generalization. Save and reload a small checkpoint as part of the same smoke test. Resolve mechanical failures while the cost of repeating the experiment is still low.

LoRA freezes the base weights and learns low-rank updates in selected modules. Its rank, scaling, dropout, target modules, and any additional trainable components belong in the experiment record. The method reduces the number of trainable parameters; it does not guarantee that a complete training workload fits a particular GPU. Context length, activations, optimizer state, and implementation still matter.[10]

Use a small, declared search over settings such as learning rate and training duration. Evaluate generated answers on Validation at chosen intervals with the same approved profile and decoding policy as the baseline. Track validation loss as a useful diagnostic, but also run the factual and instruction checks. Google’s supervised-tuning guidance separates tuning metrics from evaluation of tuned behavior; your release criteria should remain tied to the application.[3]

Imagine a checkpoint that makes the familiar introduction more polished while beginning to invent a university when asked a leading question. Its training loss may keep falling because it predicts the training responses more confidently. Your critical validation slice should reject that checkpoint. Select the candidate with the best acceptable trade-off under the rules written earlier, rather than automatically saving the last epoch as the winner.

Log unsuccessful runs too. A change that improves one language while harming another is evidence about data coverage, not merely a failed attempt to hide. When you revise examples or settings after examining Validation, record that decision and the new dataset revision. With many rounds, your judgment can become specialized to the validation set, so refreshing development coverage may be necessary. The final Test remains untouched through these choices.

Freeze the checkpoint, prompt, profile revision, preprocessing, and scoring rules, then evaluate the untuned and tuned candidates on the same Test. Use ordinary requests, boundary requests, and questions whose answers are absent from the approved evidence. Google’s Gemma guidance recommends testing unseen tasks through success, failure, and boundary cases. In this assistant, a successful response to an unknown question is an honest limitation, not a guessed biography.[11]

Pair the results by example ID. Which questions changed from wrong to right? Which changed from right to wrong? Report those transitions alongside aggregate scores and category counts. If a stochastic decoder is part of production, include repeated generations under a declared policy and preserve that variability. A single favorable answer from a handpicked seed cannot establish that the assistant reliably respects the evidence boundary.

Check retained capabilities that the product still needs: short explanations, summarization of supplied text, and the supported languages. This suite should be defined before training, rather than chosen afterward to flatter the result. A narrowly improved biography response can coexist with worse general instruction following. Evaluate the gains and regressions separately, and explain which losses would make the candidate unacceptable.

Repeat the latency and memory protocol for the actual serving configuration. If the adapter is merged, weights are quantized, or the runtime changes, evaluate that exported artifact again. Record a controlled comparison to understand the adaptation, and a production comparison to understand the endpoint users will receive. Reloading into a clean process also catches missing adapters, stale prompts, and accidental fallback to the base model.

If Test reveals a critical failure, the release can be rejected. Once you inspect those failures and use them to redesign the model, that test has become development evidence for the next iteration. Keep its historical result, and obtain a fresh held-out assessment for the revised candidate. This preserves an honest claim about unseen behavior without pretending that a failed test can never teach you anything.

A LoRA adapter is only part of the model you serve. PEFT’s checkpoint documentation distinguishes adapter weights and configuration from the pretrained base model, which is still needed to reconstruct the adapted model. Save the exact base revision along with the tokenizer and chat template. A convenient model name that later points to different files is not a sufficient identity.[12]

Attach hashes or stable versions to datasets, prompts, configuration, and evaluation files. Keep training, validation, and test IDs so someone can audit leakage or reproduce a disputed score. If resuming the exact training trajectory matters, retain the training checkpoint state as well, including optimizer, scheduler, and relevant random-number-generator state. An inference adapter export and a resumable training checkpoint serve different purposes.

Record seeds, library versions, device type, precision, and determinism settings, while stating the limits of reproducibility. PyTorch explicitly does not guarantee identical results across all releases and platforms. The practical aim is a controlled rerun with explainable differences and a reloadable release, rather than a claim that one seed makes every GPU implementation bit-for-bit identical.[13]

Perform a clean reload using only the saved bundle and documented dependencies. Ask the website-only question, an ordinary introduction, and an unknown-detail question, then run the planned acceptance suite. Record which adapter and profile revisions the process loaded. If those outputs differ from your evaluated candidate, investigate the mismatch before attaching the earlier score to the new package.

After the candidate passes, introduce it through a limited deployment with a defined observation period. For the profile assistant, you might route a small share of requests to the candidate while retaining the previous compatible release. Decide in advance what triggers a stop: a confirmed fabricated personal detail, a broken website response, an unacceptable increase in errors, or latency beyond the agreed service budget. These are proposed operating rules, not a claim that this article deploys an endpoint.

Monitor request failures, latency distributions, output lengths, and resource use at the expected concurrency. Pair operational metrics with reviewed quality samples and a recurring fixed suite of profile questions. Speed monitoring cannot tell you whether a university was invented, and a quality score cannot tell you whether visitors are timing out. Record the serving model and profile revision with each permitted diagnostic record, while keeping unnecessary personal content out of logs.

Make rollback a tested operation. Keep the previous model, adapter, tokenizer, prompt, profile compatibility, and routing configuration available; then verify that switching back actually serves the previous behavior. A filename called previous_model is not enough if its base weights are unavailable or its tokenizer no longer matches the service. Include a post-rollback check using the same critical questions.

Finally, turn reviewed production failures into the next development cycle. Refresh the approved profile when public facts change, update representative training examples when a behavior gap persists, and create new held-out cases. Keep a record of why each release was accepted. You have completed the workflow when the assistant’s improvement can be demonstrated, its artifacts can be recovered, and its behavior can be checked again after deployment.