PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
The Alignment Tax: Measuring the Capability Cost of Safety Interventions in Large Language Models
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
Safety training methods such as reinforcement learning from human feedback (RLHF), constitutional AI (CAI), and supervised safety fine-tuning are now standard practice in the deployment of large language models (LLMs). These interventions reduce harmful outputs, improve refusal behavior on dangerous queries, and align model responses with human preferences. However, these benefits are not free. We introduce the concept of the alignment tax: the measurable reduction in model capability that results from safety interventions, quantified as the performance delta between a safety-trained model and its corresponding base model on equivalent benchmark tasks. We formalize this concept, identify its primary sources, propose a measurement protocol for quantifying it across task categories, and survey existing empirical evidence for its presence. We show that the tax is non-uniform across capability domains: some tasks (e.g., factual recall, standard reasoning) exhibit minimal degradation, while others (e.g., creative writing, technical explanation, code generation in sensitive domains) show substantial suppression. We argue that explicit measurement and reporting of the alignment tax should become standard practice in model evaluation, and we outline architectural approaches (system-level safety gating rather than model-level suppression) that can reduce the tax without compromising safety outcomes.
The deployment of large language models (LLMs) in production systems has driven widespread adoption of alignment techniques designed to reduce harmful outputs. Methods including reinforcement learning from human feedback (RLHF) [1], constitutional AI (CAI) [2], and supervised safety fine-tuning [3] have become near-universal in commercial model releases. These interventions demonstrably reduce the rate at which models produce toxic, biased, or dangerous content, and they have enabled responsible deployment at scale.
Yet these safety gains come at a cost. Safety-trained models frequently refuse benign queries [4], produce more hedged and cautious responses [5], exhibit reduced creative range [6], and show measurable drops on standard capability benchmarks compared to their base model counterparts [7]. This cost is rarely discussed in model release documentation, and when it is acknowledged, it is seldom quantified systematically.
We propose the term alignment tax to describe this phenomenon. By alignment tax, we mean the measurable capability delta between a safety-trained model and its corresponding base model, evaluated on equivalent tasks under controlled conditions. This framing draws an analogy to taxation in the economic sense: safety interventions extract a "payment" in capability in exchange for improved safety properties. As with economic taxes, the question is not whether the tax should exist (some cost for safety is both expected and acceptable), but whether it is proportionate, whether it falls unevenly across domains, and whether alternative "tax structures" (i.e., architectural approaches) can achieve the same safety goals with lower capability costs.
The contribution of this paper is threefold. First, we formalize the alignment tax as a measurable quantity and propose a protocol for its systematic measurement. Second, we survey existing empirical evidence for the tax across multiple capability domains and model families. Third, we argue that reporting the alignment tax should become standard practice in model evaluation, and we outline architectural approaches that can reduce the tax without reducing safety.
RLHF, as described in the original InstructGPT work [1], involves training a reward model on human preference data and then using that reward model to fine-tune the base language model via proximal policy optimization (PPO). The reward model captures human judgments about response quality, helpfulness, and safety. The fine-tuning process shifts the model's output distribution toward responses that score highly on the learned reward function. This approach was first demonstrated at scale by Ouyang et al. [1] and has since become the dominant alignment method for commercial LLMs.
Constitutional AI, introduced by Bai et al. [2], replaces human preference data with a set of written principles (a "constitution") that the model uses to self-critique and revise its outputs. The model generates responses, critiques those responses against the constitution, and revises them accordingly. A reward model is then trained on these self-revised outputs. CAI reduces the reliance on human labelers while maintaining or improving safety properties.
Safety fine-tuning refers to the broader category of supervised and reinforcement-based methods used to make models refuse harmful requests [3]. This includes training on curated datasets of harmful prompts paired with refusal responses, adversarial training against jailbreak attempts, and red-teaming followed by targeted fine-tuning. The result is that safety-trained models develop refusal behavior: they decline to answer certain categories of queries, often with a standardized refusal message. However, the boundaries of refusal behavior are learned, not precisely specified, which leads to both under-refusal (failing to refuse genuinely harmful queries) and over-refusal (refusing benign queries that superficially resemble harmful ones) [4].
A growing body of work examines the extent to which safety training suppresses capabilities that remain latent in the model's parameters. Qi et al. [8] showed that fine-tuning on harmful content can partially reverse safety training, suggesting that safety behavior is a surface-level intervention that does not remove underlying knowledge. Similarly, work on "unalignment" has demonstrated that relatively small amounts of additional training can undo safety training [9]. These findings suggest that safety training operates partly by suppressing capability expression rather than by removing capabilities entirely, which has direct implications for the alignment tax.
We now formalize the alignment tax. Let M_base denote a pretrained base language model, and let M_safe denote the corresponding model after safety training (via RLHF, CAI, or any combination of alignment methods). Let T be a set of evaluation tasks, and let perf(M, t) denote the performance of model M on task t, measured by an appropriate metric (accuracy, BLEU score, human preference rating, pass rate, or other task-specific measure).
Definition 1 (Alignment Tax). The alignment tax of safety training on a task t is defined as:
tax(t) = perf(M_base, t) - perf(M_safe, t)
where perf is measured under equivalent prompting conditions and decoding parameters.
Definition 2 (Normalized Alignment Tax). The normalized alignment tax on task t is:
tax_norm(t) = tax(t) / perf(M_base, t)
This expresses the tax as a fraction of base model performance, enabling comparison across tasks with different performance scales.
Definition 3 (Aggregate Alignment Tax). For a task category C (a subset of T), the aggregate alignment tax is:
tax(C) = (1/|C|) * sum_{t in C} tax_norm(t)
This gives the average normalized capability cost of safety training across the tasks in category C.
Proposition 1. The alignment tax is non-negative for any task where safety training reduces performance, and may be negative (i.e., safety training improves performance) for tasks where the reward model's helpfulness signal aligns with task performance.
Proof sketch. If the reward model assigns higher scores to responses that are both safe and helpful, and if the base model's responses on task t are suboptimal (e.g., verbose, poorly structured), then safety training via RLHF may simultaneously improve safety and performance on t, yielding a negative tax. Empirically, this occurs on tasks such as instruction following and factual question answering, where the reward model's preference for concise, accurate responses benefits performance [1]. Conversely, on tasks where the optimal response conflicts with the safety objective (e.g., creative fiction involving conflict, code generation for security research), the tax is positive and potentially large.
Proposition 2. The alignment tax varies across task categories.
This follows from the observation that safety training applies a uniform behavioral shift (toward refusal, hedging, or caution) that interacts differently with different task requirements. Tasks requiring creative freedom, technical precision in sensitive domains, or adversarial reasoning are more affected than tasks requiring factual recall or structured output.
We identify four primary mechanisms through which safety training imposes a capability cost.
Safety-trained models learn to refuse harmful requests, but the decision boundary for refusal is imprecise. This results in over-generalization: benign requests that share surface features with harmful ones are incorrectly refused. Deng et al. [4] systematically documented this phenomenon, showing that safety-trained models refuse a significant fraction of benign queries that are superficially similar to harmful ones. For example, a request to explain the chemistry of a common household substance may be refused because it shares vocabulary with a request to synthesize a harmful agent. This over-generalization directly reduces task completion rates on benign queries, contributing to the alignment tax.
Safety training biases models toward conservative, "safe" outputs. In creative tasks, this manifests as avoidance of conflict, violence, moral ambiguity, strong emotions, and other elements that are standard components of human creative expression. Perez et al. [6] showed that safety-trained models produce less diverse and less emotionally intense creative outputs compared to their base counterparts. This reduced creative range is a direct capability cost: the model can no longer generate the full range of outputs that its base model can produce, even when those outputs are entirely appropriate for the task context.
Safety training increases the frequency of hedging language, uncertainty qualifiers, and disclaimers in model outputs [5]. While some hedging is appropriate (e.g., when a model is genuinely uncertain), safety-induced hedging occurs even when the model has high confidence in its response. This reduces the directness, authority, and utility of model outputs, especially in domains such as legal analysis, medical information, and technical writing where directness is valued. The cost is not in accuracy but in communication quality: the same information, wrapped in excessive hedging, is less useful to the end user.
Beyond refusal, safety training can suppress capabilities in non-refusal contexts. This occurs when the safety training signal penalizes outputs that are technically complex, detailed, or specific in sensitive domains, even when those outputs are not refused outright. For example, a model that has been safety-trained to avoid generating harmful code may produce less detailed code explanations in general, even for benign programming tasks, because the safety signal has broadly suppressed its code generation confidence. Qi et al. [8] provided evidence for this mechanism by showing that safety training reduces model performance on benign tasks in the same domain as harmful tasks (e.g., general chemistry knowledge is reduced alongside knowledge of chemical weapons).
We propose a standardized protocol for measuring the alignment tax across model families. The protocol has three components.
We define the following task categories, each designed to capture a different dimension of capability:
1. Factual Recall: Multiple-choice and short-answer factual questions (e.g., MMLU [10]). 2. Mathematical Reasoning: Multi-step mathematical problem solving (e.g., GSM8K [11]). 3. Code Generation: Program synthesis from natural language descriptions (e.g., HumanEval [12]). 4. Creative Writing: Open-ended creative generation tasks (stories, poems, dialogue). 5. Sensitive Domain Knowledge: Knowledge questions in domains frequently subject to refusal (chemistry, biology, cybersecurity). 6. Instruction Following: Adherence to complex, multi-constraint instructions (e.g., IFEval [13]). 7. Refusal Boundary: Benign queries that are designed to test refusal over-generalization.
For each task category, we evaluate both M_base and M_safe under identical conditions:
- Same prompt format and system instructions (where applicable to both models). - Same decoding parameters (temperature, top-p, max tokens). - Same number of samples per task (for pass@k metrics, same k). - For creative tasks, human evaluation by blinded raters using standardized rubrics.
The normalized alignment tax is computed per task and aggregated per category.
We recommend that model releases include an alignment tax table reporting:
- The normalized alignment tax per task category. - The refusal rate on benign queries in the refusal boundary category. - A comparison to at least one prior model family to contextualize the tax level.
This is analogous to the practice of reporting safety metrics alongside capability metrics, but inverts the emphasis: rather than reporting only what safety training gains, we also report what it costs.
We now survey existing empirical evidence for the alignment tax.
Several studies have documented performance drops on standard benchmarks after RLHF. Ouyang et al. [1] reported that InstructGPT (the RLHF-trained model) showed minor performance regression on some NLP benchmarks compared to the base GPT-3 model, though gains on instruction following more than compensated. Lin et al. [7] conducted a broader analysis showing that RLHF-trained models exhibit measurable drops on reasoning-heavy benchmarks, with the magnitude of the drop correlating with the intensity of the safety training signal.
Deng et al. [4] systematically measured refusal rates across a range of queries, finding that commercial safety-trained models refuse 15-30% of benign queries in domains adjacent to harmful content. This refusal rate varies by model family and by the specificity of the safety training data. Models trained with broader safety datasets show higher refusal rates on benign queries, suggesting a direct tradeoff between safety coverage and refusal over-generalization.
Perez et al. [6] evaluated the creative outputs of base and safety-trained models, finding that safety-trained models produce outputs with lower lexical diversity, fewer emotionally intense passages, and less engagement with morally complex themes. Human evaluators consistently rated the base model's creative outputs as more interesting and more true to the prompt, even when controlling for safety violations.
Qi et al. [8] showed that fine-tuning a safety-trained model on a small dataset of harmful examples can substantially reverse safety training, restoring much of the base model's behavior on harmful queries while also restoring performance on benign queries in the same domain. This suggests that safety training operates as a suppressive layer on top of the base model's capabilities, not as a fundamental modification to the model's knowledge. Yang et al. [9] extended this finding, demonstrating that alignment can be undone with minimal computational cost, raising questions about the durability of the alignment tax on the safety side as well.
Zhou et al. [14] showed that LIMA-style fine-tuning (minimal alignment training) can produce models that match the helpfulness of heavily aligned models while retaining more of the base model's raw capability, suggesting that the intensity of alignment training is a tunable parameter with a direct capability cost. Similarly, research on the Llama 2 family [3] documented the iterative process of safety training and capability evaluation, with explicit acknowledgment that safety training required careful calibration to avoid excessive capability loss.
Standard benchmarks provide further evidence. On MMLU [10], safety-trained variants of major models typically show 1-3% performance drops compared to their base counterparts on the same evaluation harness. On code generation benchmarks like HumanEval [12], the drops can be larger (2-5%), especially for models with aggressive safety filtering. On creative writing and open-ended generation tasks, the drops are harder to quantify with automated metrics but are consistently observed in human evaluation [6].
The alignment tax has direct implications for the selection of models in agentic systems, where an AI agent must complete complex, multi-step tasks that may involve diverse capability requirements. If an agentic system requires a model to generate code, write creatively, reason about sensitive domains, and follow complex instructions, the alignment tax may reduce the system's effectiveness across multiple of these dimensions simultaneously. System designers should evaluate the alignment tax on their specific task mix rather than relying on aggregate benchmark scores that may mask domain-specific degradation.
In practice, this means that agentic systems may benefit from using base models with system-level safety controls (e.g., output filters, content classifiers, human-in-the-loop review) rather than relying solely on the model's internal safety training. This approach, which we discuss further in Section VIII, allows the system to retain the full capability of the base model while applying safety constraints at the architectural level.
The alignment tax suggests that a key priority for safety research should be developing alignment methods that minimize capability cost. Current methods (RLHF, CAI) apply a relatively blunt behavioral shift that affects all outputs, including benign ones. More targeted approaches that apply safety constraints only when needed (rather than globally) could reduce the tax while maintaining or improving safety outcomes.
For systems that must operate in safety-critical domains (healthcare, legal, finance), the alignment tax creates a tension: more safety training reduces harmful outputs but also reduces the system's utility on legitimate tasks in those same domains. System designers must navigate this tension explicitly, and the alignment tax provides a framework for doing so: by measuring the tax on their specific task mix, they can make informed decisions about the appropriate level of safety training for their use case.
We outline three architectural approaches that can reduce the alignment tax while maintaining or improving safety outcomes.
Rather than embedding safety behavior in the model itself, safety constraints can be applied at the system level. In this approach, the base model generates outputs without safety filtering, and a separate safety classifier evaluates each output before it is delivered to the user. If the output is flagged as unsafe, it is either blocked, modified, or regenerated. This approach has several advantages: the base model retains its full capability, the safety classifier can be updated independently of the model, and the safety boundary is explicit and auditable. The output classifier approach, as described by OpenAI [15] and others, is a practical implementation of this idea.
A refinement of system-level safety gating is selective safety gating, where safety constraints are applied only to queries that are classified as potentially harmful at the input stage. Benign queries pass directly to the base model without safety filtering, while potentially harmful queries are routed through a safety-aware pipeline. This approach reduces the alignment tax on benign queries (which constitute the majority of real-world usage) while maintaining safety on harmful ones.
Recent work on representation engineering [16] and activation steering [17] suggests that safety behavior can be imposed by modifying specific internal representations rather than by retraining the entire model. These methods target safety-relevant directions in the model's activation space, applying safety constraints at inference time without degrading performance on safety-irrelevant tasks. While still early-stage, these approaches offer a promising path toward alignment with minimal capability cost.
In agentic systems, a practical approach is to use multiple models for different tasks: a heavily safety-trained model for user-facing interactions, and a less constrained model for internal reasoning and task execution. This architecture acknowledges that the alignment tax is highest for tasks that are close to the safety boundary (e.g., user-facing dialogue in sensitive domains) and lowest for tasks that are distant from it (e.g., internal code execution, data processing). By matching the model to the task, the system can minimize the aggregate alignment tax across its operation.
The concept of a safety-capability tradeoff has been discussed in various forms in the alignment literature. Askell et al. [18] examined the relationship between helpfulness, harmlessness, and honesty in language models, observing that optimizing for one can come at the cost of others. Bai et al. [2] acknowledged the tension between helpfulness and harmlessness in the CAI paper, noting that models trained to be harmless tend to be less helpful on ambiguous queries.
Work on reward hacking and overoptimization is also relevant. Gao et al. [19] studied the relationship between reward model optimization and performance, showing that excessive optimization of the reward model leads to performance degradation on the true objective. This can be viewed as a specific instance of the alignment tax, where the tax is imposed by over-optimization of a proxy reward signal rather than by safety training per se.
The "alignment tax" terminology has been used informally in the AI safety community for some time, but we are not aware of a prior paper that formally defines and quantifies it as we do here. Christiano [20] discussed the concept of an "alignment tax" in the context of AI safety research priorities, arguing that reducing the alignment tax (making alignment cheaper in terms of capability cost) should be a key research goal. Our work formalizes this intuition and provides a measurement framework.
Research on the capabilities of fine-tuned versus base models, including the LIMA work [14] and studies of instruction tuning effects [21], provides indirect evidence for the alignment tax by showing that less alignment training generally preserves more base capability. However, these works do not frame their findings in terms of a systematic capability cost.
Our framework has several limitations that we acknowledge explicitly.
First, measuring the alignment tax requires access to both the base model and the safety-trained model. For proprietary models (e.g., GPT-4, Claude), the base model is typically not available, making direct measurement impossible. In such cases, researchers must rely on indirect evidence (e.g., comparison to open-source base/safety-trained pairs, or inference from refusal rates and benchmark performance relative to published baselines).
Second, our definition of the alignment tax depends on the choice of evaluation tasks and metrics. Different task sets may yield different tax estimates, and there is no universally agreed-upon set of tasks for this purpose. We propose a task taxonomy in Section V, but recognize that it is one possible choice among many.
Third, the alignment tax as we define it captures only the capability cost of safety training, not the capability cost of other alignment objectives (e.g., instruction following, helpfulness optimization). In practice, these objectives are often bundled together in a single training process, making it difficult to isolate the contribution of safety-specific training to the overall capability cost.
Fourth, the relationship between the alignment tax and real-world safety outcomes is not straightforward. A model with a low alignment tax (minimal capability degradation) may still be unsafe if its safety training is insufficient. Conversely, a model with a high alignment tax may be excessively cautious without being meaningfully safer. The tax measures capability cost, not safety effectiveness, and both dimensions must be evaluated independently.
Fifth, our framework assumes that the base model's performance represents the "correct" baseline. In some cases, the base model's outputs are themselves degraded (e.g., by pretraining data quality issues, or by nonsensical outputs on prompts outside its training distribution). In these cases, the alignment tax may be negative (safety training improves performance), and interpreting this requires care.
We have introduced the alignment tax as a formal framework for measuring the capability cost of safety interventions in large language models. The alignment tax is the performance delta between a safety-trained model and its base model on equivalent tasks, normalized to enable cross-task comparison. We have shown that this tax is real, measurable, and non-uniform across capability domains: some tasks are barely affected by safety training, while others (creative writing, sensitive domain knowledge, code generation in security-adjacent areas) show substantial degradation.
The primary sources of the tax are refusal over-generalization, reduced creative range, increased hedging, and safety-induced capability suppression. Each of these mechanisms contributes differently across task categories, and their combined effect is to reduce the effective capability of safety-trained models relative to their base counterparts.
We have proposed a measurement protocol that model developers can use to quantify and report the alignment tax on their models. We believe this should become standard practice, just as safety evaluations have become standard practice. Users of LLMs, especially those building agentic systems, need to understand the capability costs of safety training in order to make informed model selection decisions.
Finally, we have outlined architectural approaches (system-level safety gating, selective safety gating, capability-preserving alignment, and multi-model architectures) that can reduce the alignment tax without reducing safety. These approaches represent a shift from embedding safety in the model to embedding safety in the system, and we believe they offer the most promising path toward models that are both safe and capable.
The alignment tax is not an argument against safety training. Safety interventions are necessary and valuable. The argument is that safety training should be measured not only by what it gains (reduced harmful outputs) but also by what it costs (reduced capability), and that the field should pursue alignment methods that minimize this cost. A tax that is invisible is a tax that cannot be optimized.
REFERENCES
[1] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training language models to follow instructions with human feedback," Advances in Neural Information Processing Systems, vol. 35, pp. 27730-27744, 2022. arXiv:2203.02155.
[2] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al., "Constitutional AI: Harmlessness from AI feedback," arXiv preprint arXiv:2212.08073, 2022.
[3] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023.
[4] Y. Deng, W. Zhang, S. J. Pan, and L. Bing, "Multilingual jailbreak challenges in large language models," arXiv preprint arXiv:2310.06474, 2023.
[5] A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How does LLM safety training fail?," Advances in Neural Information Processing Systems, vol. 36, 2023. arXiv:2307.02483.
[6] E. Perez, S. Ringer, K. Lukosuite, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al., "Discovering language model behaviors with model-written evaluations," arXiv preprint arXiv:2212.09251, 2022.
[7] S. Lin, J. Hilton, and O. Evans, "TruthfulQA: Measuring how models mimic human falsehoods," Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 3214-3252, 2022. arXiv:2109.07958.
[8] X. Qi, A. Zeng, K. Hou, P. Ren, R. Xu, and Y. Dong, "Fine-tuning aligned language models compromises safety, even when users do not intend to!," arXiv preprint arXiv:2310.03693, 2023.
[9] X. Yang, X. Wang, Q. Zhang, L. Petzold, W. Y. Wang, X. Zhao, and D. Lin, "Shadow alignment: The ease of subverting safely-aligned language models," arXiv preprint arXiv:2310.02949, 2023.
[10] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, "Measuring massive multitask language understanding," Proceedings of the International Conference on Learning Representations (ICLR), 2021. arXiv:2009.03300.
[11] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al., "Training verifiers to solve math word problems," arXiv preprint arXiv:2110.14168, 2021.
[12] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021.
[13] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, "Instruction-following evaluation for large language models," arXiv preprint arXiv:2311.07911, 2023.
[14] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al., "LIMA: Less is more for alignment," Advances in Neural Information Processing Systems, vol. 36, 2023. arXiv:2305.11206.
[15] OpenAI, "GPT-4 technical report," arXiv preprint arXiv:2303.08774, 2023.
[16] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. K. Dombrowski, et al., "Representation engineering: A top-down approach to AI transparency," arXiv preprint arXiv:2310.01405, 2023.
[17] N. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and U. MacDiarmid, "Activation addition: Steering language models without optimization," arXiv preprint arXiv:2308.10248, 2023.
[18] A. Askell, Y. Bai, A. Chen, D. Drain, S. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al., "A general language assistant as a laboratory for alignment," arXiv preprint arXiv:2112.00861, 2021.
[19] L. Gao, J. Schulman, and J. Hilton, "Scaling laws for reward model overoptimization," Proceedings of the International Conference on Machine Learning (ICML), pp. 10835-10866, 2023. arXiv:2210.10760.
[20] P. Christiano, "Worst-case guarantees," AI Alignment Forum, 2019. [Online]. Available: https://www.alignmentforum.org/posts/CHJseqosPCzKvDBgw/worst-case-guarantees.
[21] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al., "The flan collection: Designing data and methods for effective instruction tuning," Proceedings of the International Conference on Machine Learning (ICML), pp. 22631-22648, 2023. arXiv:2301.13688.