Instruction tuning uses examples of tasks expressed as instructions, together with target responses. It is commonly implemented as supervised fine-tuning, or SFT. For a causal language model, the loss rewards predicting the target tokens in their demonstrated context. Instruction tuning describes the content and purpose of the training data; it does not require updating every weight. You can apply the same instruction dataset through full fine-tuning, LoRA, or another compatible adaptation method.[1][20]

For the returns assistant, a training example contains the policy, order facts, user request, and a correct response. During teacher-forced training, the next target token is predicted using preceding target tokens. In an answer-only setup, prompt and padding positions are excluded from the loss; assistant answer tokens remain supervised. Verify the actual tokenized sequence and chat template before trusting that split. Current SFT tooling exposes completion-only and assistant-only loss options, but their applicability depends on dataset format and template support.[20]

A useful dataset needs more than many variations of order A17. Include items outside the window, different policy windows, missing delivery information, used items, and requests to promise a refund. Preserve the same decision logic while changing wording. If every example says “20 days,” the model can imitate a constant instead of learning to use the provided facts. Keep related conversations and near-duplicate templates out of the test split so that the evaluation asks something genuinely new.

The main limitation is that imitation rewards the demonstrated answer, including its mistakes. A polished response with an unsupported promise is still a bad target. Inspect a small batch end to end: rendered prompt, target, token boundaries, loss-bearing positions, and generated response after training. Falling training loss only tells you that the selected targets are becoming easier to predict. It does not establish that the assistant handles unseen policies or respects the boundary between advice and an executed refund.

LoRA, Low-Rank Adaptation, keeps a base weight matrix fixed and learns an additive update expressed through two smaller matrices. For W with output dimension d_out and input dimension d_in, write the update as BA, where A has shape r × d_in and B has shape d_out × r. The rank r controls the update’s maximum rank. The usual formulation also scales the update by α/r. LoRA changes the representation of the trainable update; it does not supply a training objective or a dataset.[2][16]

Suppose you adapt one 4,096 × 4,096 projection inside the returns assistant with rank 8. The original matrix contains 16,777,216 parameters. The two update matrices contain 8 × (4,096 + 4,096) = 65,536, or about 0.391% as many. That calculation is exact for this chosen layer. A real model’s total depends on which projections you target, their shapes, the ranks, and any additional trainable parameters. “Two small matrices per layer” is therefore a shorthand for each adapted target matrix.

You might use those updates to make the assistant follow the response schema more consistently. Start with a limited, measurable failure, then compare ranks and target modules against the same held-out cases. A larger rank gives the update more freedom, but also more parameters to fit and more room to learn accidental patterns. Report the quality result separately from the parameter calculation; a small update can be useful without matching full fine-tuning on every task.

For an ordinary compatible linear layer, a LoRA update can be merged into the base weight so the separate adapter path is no longer required at inference. Keeping adapters separate can instead support switching tasks or customers. Quantized weights, supported layer types, and the serving engine complicate that choice. Check the merged or adapter-loaded artifact you will actually serve. Avoid treating the algebraic possibility of merging as a guarantee of zero overhead for every deployment.[16]

QLoRA combines a frozen quantized base with trainable low-rank adapters. The original work uses 4-bit NormalFloat, double quantization of quantization constants, and paged optimizers to address memory usage and spikes. Computation still uses suitable higher-precision arithmetic where required, and gradients train the adapters rather than directly updating the stored 4-bit base weights. Calling this “training a model in four bits” hides the distinction between storage precision, compute precision, and the parameters receiving updates.[3]

For a hypothetical seven-billion-parameter base, two bytes per parameter means 14 GB of raw 16-bit weight storage. Four bits per parameter means 3.5 GB before scales and other metadata. These decimal-GB estimates exclude activations, gradients, optimizer state, temporary buffers, adapters, and any extra models. They explain why quantizing the base helps, but they are not a prediction of total GPU memory. Longer conversations in the returns dataset can change the practical budget substantially.

The QLoRA paper demonstrated a 65-billion-parameter model on a single 48 GB GPU under its reported setup. That is a result with specified conditions, not a promise that your model fits your card. For the support assistant, measure a representative training step at the intended sequence length and batch configuration. Then test generation from the saved artifact. A successful load or a small adapter file is not evidence that training fits, converges, or produces a deployable model.[3]

Prefix tuning learns continuous task-specific vectors that the frozen model can attend to. In Transformer implementations these commonly become added key/value states at multiple attention layers. They are learned numerical representations, not a natural-language instruction you type at the beginning of a chat. The original method also uses a parameterization designed to make optimizing those prefix representations easier. The essential distinction is that task adaptation lives in added attention context rather than an additive update to W.[4][17]

For the returns assistant, imagine training a prefix that consistently steers the model toward the requested support format while the explicit policy and order facts remain normal input. The prefix learns through the downstream task loss. It does not give the system new access to the order database, and it does not make a changing policy current. Its influence must still be tested when the conversation is longer, when required facts are absent, and when the user asks for an unauthorized action.

A small prefix checkpoint can be attractive when you want many task-specific adaptations. The serving path must, however, support injecting those learned states, and the extra attention positions can add cache and computation costs. Measure those costs with your actual prompt lengths. Do not compare prefix parameter counts directly with LoRA counts and conclude that the smaller artifact will always be faster. Where the parameters enter the network matters as much as how many are stored.

In the classic bottleneck-adapter approach, you insert small modules into the pretrained network. A module projects a hidden representation down to a smaller dimension, applies a nonlinearity, and projects back up, often using a residual connection. The main pretrained parameters remain fixed, while adapter parameters and, in some configurations, selected normalization parameters are trained. Placement differs across designs. “Between layers” is a loose description; the original Transformer adaptation inserts modules within layer substructure.[5]

Applied to the support assistant, these modules learn transformations that help map the provided order and policy into the desired response. Their bottleneck dimension governs part of their capacity and storage cost. Unlike standard LoRA’s linear additive update, a bottleneck adapter with a nonlinearity generally cannot be folded into one unchanged base linear operation. That structural difference is why inference overhead needs its own measurement rather than being inferred from a tiny checkpoint.

The word “adapter” is also used broadly for LoRA checkpoints and other PEFT methods. When someone proposes adapter tuning, ask what module is actually being trained and where it runs. For this article, the term means the classic inserted bottleneck modules. That vocabulary prevents a practical misunderstanding: a serving engine that loads LoRA adapters does not automatically support every architecture also called an adapter.

P-tuning learns continuous prompt embeddings, typically through a prompt encoder, instead of relying only on hand-written discrete prompt tokens. In the original approach, trainable prompt representations can appear within the input sequence. Implementations can use an MLP or recurrent encoder; describing every version as “a small MLP on a frozen base” is too narrow. The original research studies both frozen and tuned language models, so freezing is a configuration choice that should be stated.[6][17]

P-tuning v2 is a later deep-prompt-tuning approach adapted for a broader set of natural-language-understanding settings, with prompts across layers. It should not be silently treated as identical to the original input-level method. Prefix tuning, basic soft-prompt tuning, and P-tuning share the idea of learned continuous context, but differ in insertion points, parameterization, and the settings studied. A paper’s empirical comparison is evidence about that setting, not a universal ranking of these families.[7]

For the returns assistant, the candidate adaptation might learn a prompt representation for the support task while keeping current policy text explicit. Check whether your training library and serving stack implement the same variant. If an experiment trains deep per-layer prompts and deployment later supplies only input embeddings, you have changed the computation. A clear artifact record should include the prompt method, base checkpoint, insertion configuration, and evaluation results.

BitFit restricts fine-tuning to bias terms, or a subset of them. For a transformation written as Wx + b, the basic idea is to keep W fixed while allowing selected b values to change. The published study focuses on pretrained BERT models and evaluates the trade-off across different data regimes. Its results establish a useful sparse adaptation baseline, not a guarantee that bias-only training is sufficient for every generative language model.[8]

The trainable fraction is architecture-dependent. A figure such as 0.08% is not a definition of BitFit, and some model components have no bias at all. Before proposing it for the returns assistant, inspect which parameters actually qualify and whether any other task-specific head is being trained. If the chosen architecture offers very few suitable bias terms, the method may have little capacity to change the behavior you care about.

A bias-only baseline can answer a useful experimental question: is this support task mostly exposing a capability the base already has, or does it need a more expressive update? Compare it with a prompt baseline and a small LoRA run on identical held-out cases. The smallest trainable fraction is not automatically the best choice when the resulting assistant still produces invalid fields or unsupported promises.

RLHF means reinforcement learning from human feedback. A common pipeline starts with supervised instruction tuning, collects human preferences over candidate responses, fits a reward model, and then optimizes the language-model policy against that reward. PPO is a well-known optimizer used in this pattern, with a value estimator and controls intended to limit disruptive policy changes. The acronym identifies the role of human feedback; it does not make PPO the only possible algorithm.[9]

For the returns assistant, the preference above rewards staying within the system’s authority. Both responses get the arithmetic right, so the comparison isolates a different failure: claiming an action that never happened. Human reviewers need the actual policy, order context, tool state, and a rubric. Otherwise a confident unsupported response can look more helpful than a careful correct one. Include cases where the preferred answer asks for missing information, rather than always rewarding a more immediate answer.

RLAIF replaces some human-provided judgments with AI-generated feedback. Constitutional AI is one documented approach: model-generated critiques and revisions support a supervised phase, and AI preferences support reward-model training for an RL phase. The people defining the principles, judge prompt, data, and evaluation still shape the result. AI feedback can also populate a preference dataset used by a method such as DPO; the feedback source and the optimizer remain separate choices.[10][11]

For our assistant, an AI judge could apply the same “no invented refund” rubric, but its decisions need calibration against reviewed examples. Judge errors can become repeated training signals. Evaluate disagreement cases and avoid allowing a candidate answer to rewrite the judging instructions. Model-based review can reduce some labeling work; its cost and reliability depend on the task. A high reward score is evidence that the policy satisfies that reward system, not a universal certificate of helpfulness or factual accuracy.

Direct Preference Optimization trains on a prompt and two responses, one preferred over the other. In its standard form, the loss increases the preferred response’s relative likelihood compared with the rejected response, measured against a fixed reference policy. The derivation connects that objective to a particular reward-model parameterization. You do not need to fit a separate explicit reward model or run the standard online PPO rollout loop for each training batch. You still perform ordinary gradient-based optimization of the policy.[11][18]

Read the formula using the two support responses. The current model and reference each assign a probability to each complete answer under the same prompt. DPO compares how the chosen-versus-rejected preference shifts relative to the reference. “Closed-form objective” does not mean the final model weights have a closed-form solution. The loss gives you a computable training target; repeated batches and gradient updates still do the learning.[11][18]

DPO can be an economical next experiment when you have trustworthy pairs and the assistant already produces plausible responses. You may train only LoRA parameters while using DPO, because the objective and update mechanism are compatible choices. Keep candidate responses attached to exactly the context they were judged against. A response rejected under a 30-day policy may be correct under another policy. Also check whether the preference labels accidentally reward answer length or familiar phrasing instead of the intended behavior.

The important evaluation remains external to the pair loss: unseen policies, JSON validity, arithmetic, appropriate clarification, and invented-action rate. A dataset of weak or contradictory comparisons can teach a consistent but unwanted preference. Reference-model storage and computing both candidates also have costs. Some implementations can precompute fixed reference log-probabilities; that reduces a resource requirement without removing the reference policy’s conceptual role.[18]

GRPO, Group Relative Policy Optimization, was introduced in DeepSeekMath. It samples multiple outputs for a prompt, scores them, and uses their group statistics to construct advantages. Compared with the familiar PPO setup, it avoids a separately learned critic or value network for that estimate. It still needs a policy, generated responses, and a source of rewards. The original method also includes policy-ratio clipping and a KL-related regularization term; an advantage formula alone is not the complete training algorithm.[12]

In this deliberately small example, a verifier checks the structured result for order A17. Two outputs are correct, and two violate the arithmetic or allowed next action. The reward vector is [1, 1, 0, 0], with mean 0.5. Using the population standard deviation gives 0.5. Subtracting the mean and dividing by that deviation produces [+1, +1, −1, −1]. These values explain the relative signal; they are not reported training results or claims about a real model.

The contrast within a group matters. If every response fails and receives the same zero score, a stabilized group-relative calculation supplies no task-reward ranking among them. A KL term can still affect an update, but it does not tell the model which failed answer was closer to success. You may need easier training cases, a more informative valid reward, or a stronger starting policy. Silently treating identical rewards as evidence of learning would hide a failure in the training signal.[12][19]

GRPO saves the cost of a separate critic in this comparison, yet producing several responses per prompt can be expensive. Account for generation length, group size, reward evaluation, and policy updates. For a simple support schema, supervised examples or preference pairs might already address the problem. A group-based RL loop becomes a candidate when generating and checking alternatives supplies useful information that those simpler experiments have not captured.

Reinforcement learning with verifiable rewards uses checkable task outcomes as a source of reward. Examples include matching a known mathematical answer or passing a specified set of code tests. DeepSeek-R1 reports rule-based rewards for suitable reasoning tasks, alongside other training stages and reward choices. RLVR describes where reward evidence comes from. GRPO describes an optimization approach. You can therefore use GRPO with verifiable rewards; choosing one does not rule out the other.[13]

For the returns assistant, a narrow verifier can parse the JSON, require the agreed fields and types, calculate 30 − 10, check the unused-item condition, and accept only the permitted next-step value. Keep the policy inputs external to the generated answer so the model cannot make its own output correct by rewriting the rule. Under this toy contract, a valid response earns one and a failure earns zero. Verifiable rewards can also be graded or combine several checks; binary reward is one design, not the definition of RLVR.

The verifier’s scope is the central limitation. Valid JSON does not establish that a natural-language explanation is empathetic or complete. Passing a finite set of program tests does not prove every possible execution correct. Even exact arithmetic is useful only if the elapsed-day input and policy interpretation are correct. Separate those sources of evidence and test adversarial outputs that satisfy the superficial format while violating the intended task.

For this particular arithmetic, an ordinary function is a better production source of truth than asking the language model to become a calculator. The RLVR example teaches the reward mechanism; it is not a claim that reinforcement learning is necessary for this product. You would justify an RL experiment by a broader class of useful, verifiable behaviors and compare it with a system that simply calls the reliable calculation. Prefer an improvement you can measure over an additional training stage you can merely name.

Federated learning coordinates training across participants that retain their local data and share model updates for aggregation. FedAvg is a foundational example based on repeated local optimization and model averaging. Federated fine-tuning applies this style of coordination to adaptation of a pretrained model. It is a deployment and optimization arrangement, not a substitute for supervised learning, preferences, or parameter-efficient updates. Local training still needs an explicit objective.[14]

Imagine three regional branches operating the same returns assistant, with support conversations that cannot simply be placed in one shared training table. Each branch can train from an agreed starting checkpoint using its permitted local records. A coordinator receives compatible updates, aggregates them according to the chosen protocol, and evaluates a candidate shared model. The branches can have different language mixtures, policy windows, and volumes. Aggregate improvements can hide a regression at a smaller branch, so evaluate locally as well as globally.

Training LoRA modules locally can reduce the number of transmitted parameters, but aggregation needs care. Simply averaging A matrices and averaging B matrices is not generally the same as averaging their products BA. Different ranks, target modules, base checkpoints, or factor representations make compatibility more difficult. Specify what is aggregated and why that operation corresponds to the intended update. “We only exchange adapters” is a description of transport size, not a complete aggregation algorithm.

Keeping raw records local also does not establish that updates reveal nothing about them. Research on gradient leakage demonstrates that shared gradients can disclose training information under studied conditions. Privacy and security therefore require an explicit threat model and appropriate controls, potentially including secure aggregation or differential privacy, with their own assumptions and costs. Federated training has additional operational issues too: unreliable participants, poisoned updates, uneven hardware, and communication delays. Choose it when the data-ownership constraint warrants that complexity.[15]

For the fictional assistant, begin by collecting representative requests and documenting the failure categories. Test the untuned model with the intended prompt, current policy context, structured-output checks, and deterministic arithmetic. Save the actual outputs. If the main problem is inconsistent response structure or handling of missing facts, a carefully curated SFT dataset is a reasonable next experiment. If the remaining issue is a preference between plausible responses, reviewed chosen/rejected pairs may support a DPO experiment.

LoRA or QLoRA can be candidates for either experiment when their supported architecture and memory behavior fit the setup. Keep the base model, data split, prompt template, and decoding settings stable enough for a meaningful comparison. Record target modules, rank, precision choices, training seed, learning rate, and the checkpoint selected by validation. These details turn “LoRA worked” into a result another person can reproduce and investigate.

Before a long run, perform a small operational check: confirm that the intended parameters are trainable, inspect loss-bearing tokens, run forward and backward passes, check finite loss and gradients, and save and reload a checkpoint. Then run the full evaluation on the reloaded artifact. These checks answer different questions. A finite loss establishes basic numerical behavior; a successful reload establishes artifact compatibility; held-out task results establish whether the proposed adaptation helped.

For the support task, report schema validity, correct policy application, arithmetic accuracy, missing-information handling, and unsupported-action claims separately. Also measure latency and memory in the serving configuration, and check that unrelated useful behavior has not degraded. Group the evaluation by policy variant and branch so that a strong average cannot conceal a local failure. Do not select the winning checkpoint on the final test set and then report that same set as an independent estimate.

The useful outcome is a justified combination, not a winner among twelve labels. You may end up with instruction data, a quantized frozen base, LoRA updates, and a later preference-learning experiment. Another setting may justify deep prompts or federated coordination. Keep asking the same four questions: what should the assistant learn, which evidence teaches it, what can be updated affordably, and where can training occur? The method names become easier to use once each has a specific job.