ARCHIVES
Year 2026 · Volume 5 · Issue 2
Case Study
Explicit Content Generation in Text-to-Image Models: Training Mechanisms, Moderation Failures, and Safety Limits
Yajnesh Rao1
1 Independent Researcher, Data Engineer Karnataka, India.
Published Online: May-August 2026
Pages: 1005-1020
Cite this article
↗ https://www.doi.org/10.59256/indjcst.20260502111References
1. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti,
2. T. Coombes, A. Katta, C. Mullis, and M. Wortsman, “Laion-5b: An open large-scale dataset for training next generation image-
text models,” arXiv preprint arXiv:2210.08402, 2022.
3. S. Y. Gadre et al., “Datacomp: In search of the next generation of multimodal datasets,” in Advances in Neural Information
Processing Systems, 2023.
4. OpenAI, “Dall·e 2 pre-training mitigations,” https://openai.com/index/ dall-e-2-pre-training-
mitigations/, 2022, accessed: 2026-02-13.
5. “Stable diffusion v1-5 model card,” https://huggingface.co/stable-diffusion-v1-5/ stable-diffusion-v1-5, accessed: 2026-02-13.
6. OpenAI, “Dall·e 3 system card,” https://openai.com/index/dall-e-3-system-card/, 2023, accessed: 2026-02-13.
7. ——, “Openai moderation guide,” https://platform.openai.com/docs/guides/ moderation, accessed: 2026-02-13.
8. “Stable diffusion safety checker model card,” https://huggingface.co/CompVis/ stable-diffusion-safety-checker, accessed: 2026-02-
13.
9. “Stable diffusion v1-4 model card,” https://huggingface.co/CompVis/ stable-diffusion-v1-4, accessed: 2026-02-13.
10. J. Rando, D. Paleka, D. Lindner, L. Heim, and F. Tr a m`er , “Red-teaming the stable diffusion safety filter,” arXiv preprint
arXiv:2210.04610, 2022.
11. P. Christiano et al., “Deep reinforcement learning from human preferences,” NeurIPS, 2017.
12. OpenAI, “Openai usage policies,” https://openai.com/policies/usage-policies/, 2025.
13. ——, “Openai model spec,” https://model-spec.openai.com/2025-02-12.html, 2025.
14. L. Ahmad et al., “Openai’s approach to external red teaming,” https://cdn.openai. com/papers/openais-approach-to-external-red-
teaming.pdf, 2024.
15. Z. Ba et al., “Surrogateprompt: Bypassing safety filters via substitution,” arXiv preprint arXiv:2309.14122, 2023.
16. Y. Deng and H. Chen, “Divide-and-conquer attack on text-to-image models,” arXiv preprint arXiv:2312.07130, 2023.
17. Z. Jiang et al., “Jailbreaking safeguarded text-to-image models,” arXiv preprint arXiv:2503.01839, 2025.
18. J. Ma et al., “Jailbreaking prompt attack,” arXiv preprint arXiv:2404.02928, 2024.
19. Y. Yang et al., “Defending text-to-image models from adversarial prompts,” in NeurIPS, 2024.
20. T. Xie et al., “Context shifts in nsfw moderation,” https://openreview.net/forum? id=KsxSp2LMlo, 2024.
21. L. Li et al., “T2isafety benchmark,” arXiv preprint arXiv:2501.12612, 2025.
22. Z. Cheng et al., “Overt: Over-refusal evaluation,” arXiv preprint arXiv:2505.21347, 2025.
23. “Mitigating oversexualization,” https://openreview.net/forum?id=FxX6g2aQZO, 2024.
24. V. Vapnik, Statistical Learning Theory. Wiley, 1998.
25. T. Cover and J. Thomas, Elements of Information Theory. Wiley, 2006.
26. I. Goodfellow et al., “Explaining and harnessing adversarial examples,” ICLR, 2015.
27. “European union artificial intelligence act,” https://digital-strategy.ec.europa.eu/ en/policies/european-approach-artificial-intelligence,
2024.
28. “Uk online safety act,” https://www.legislation.gov.uk/ukpga/2023/50/contents, 2023.
29. “Oecd ai principles,” https://oecd.ai/en/ai-principles, 2019.
30. “Ai policy for children,” https://www.unicef.org/globalinsight/reports/ policy-guidance-ai-children, 2021.
31. “Ethics of artificial intelligence,” https://www.unesco.org/en/artificial-intelligence/ recommendation-ethics, 2021.
32. “General data protection regulation,” https://eur-lex.europa.eu/eli/reg/2016/679/ oj, 2016.
33. “Ftc ai and impersonation guidance,” https://www.ftc.gov/business-guidance/ ai-tools, 2024.
34. E. Bender et al., “On the dangers of stochastic parrots,” FAccT, 2021.
2. T. Coombes, A. Katta, C. Mullis, and M. Wortsman, “Laion-5b: An open large-scale dataset for training next generation image-
text models,” arXiv preprint arXiv:2210.08402, 2022.
3. S. Y. Gadre et al., “Datacomp: In search of the next generation of multimodal datasets,” in Advances in Neural Information
Processing Systems, 2023.
4. OpenAI, “Dall·e 2 pre-training mitigations,” https://openai.com/index/ dall-e-2-pre-training-
mitigations/, 2022, accessed: 2026-02-13.
5. “Stable diffusion v1-5 model card,” https://huggingface.co/stable-diffusion-v1-5/ stable-diffusion-v1-5, accessed: 2026-02-13.
6. OpenAI, “Dall·e 3 system card,” https://openai.com/index/dall-e-3-system-card/, 2023, accessed: 2026-02-13.
7. ——, “Openai moderation guide,” https://platform.openai.com/docs/guides/ moderation, accessed: 2026-02-13.
8. “Stable diffusion safety checker model card,” https://huggingface.co/CompVis/ stable-diffusion-safety-checker, accessed: 2026-02-
13.
9. “Stable diffusion v1-4 model card,” https://huggingface.co/CompVis/ stable-diffusion-v1-4, accessed: 2026-02-13.
10. J. Rando, D. Paleka, D. Lindner, L. Heim, and F. Tr a m`er , “Red-teaming the stable diffusion safety filter,” arXiv preprint
arXiv:2210.04610, 2022.
11. P. Christiano et al., “Deep reinforcement learning from human preferences,” NeurIPS, 2017.
12. OpenAI, “Openai usage policies,” https://openai.com/policies/usage-policies/, 2025.
13. ——, “Openai model spec,” https://model-spec.openai.com/2025-02-12.html, 2025.
14. L. Ahmad et al., “Openai’s approach to external red teaming,” https://cdn.openai. com/papers/openais-approach-to-external-red-
teaming.pdf, 2024.
15. Z. Ba et al., “Surrogateprompt: Bypassing safety filters via substitution,” arXiv preprint arXiv:2309.14122, 2023.
16. Y. Deng and H. Chen, “Divide-and-conquer attack on text-to-image models,” arXiv preprint arXiv:2312.07130, 2023.
17. Z. Jiang et al., “Jailbreaking safeguarded text-to-image models,” arXiv preprint arXiv:2503.01839, 2025.
18. J. Ma et al., “Jailbreaking prompt attack,” arXiv preprint arXiv:2404.02928, 2024.
19. Y. Yang et al., “Defending text-to-image models from adversarial prompts,” in NeurIPS, 2024.
20. T. Xie et al., “Context shifts in nsfw moderation,” https://openreview.net/forum? id=KsxSp2LMlo, 2024.
21. L. Li et al., “T2isafety benchmark,” arXiv preprint arXiv:2501.12612, 2025.
22. Z. Cheng et al., “Overt: Over-refusal evaluation,” arXiv preprint arXiv:2505.21347, 2025.
23. “Mitigating oversexualization,” https://openreview.net/forum?id=FxX6g2aQZO, 2024.
24. V. Vapnik, Statistical Learning Theory. Wiley, 1998.
25. T. Cover and J. Thomas, Elements of Information Theory. Wiley, 2006.
26. I. Goodfellow et al., “Explaining and harnessing adversarial examples,” ICLR, 2015.
27. “European union artificial intelligence act,” https://digital-strategy.ec.europa.eu/ en/policies/european-approach-artificial-intelligence,
2024.
28. “Uk online safety act,” https://www.legislation.gov.uk/ukpga/2023/50/contents, 2023.
29. “Oecd ai principles,” https://oecd.ai/en/ai-principles, 2019.
30. “Ai policy for children,” https://www.unicef.org/globalinsight/reports/ policy-guidance-ai-children, 2021.
31. “Ethics of artificial intelligence,” https://www.unesco.org/en/artificial-intelligence/ recommendation-ethics, 2021.
32. “General data protection regulation,” https://eur-lex.europa.eu/eli/reg/2016/679/ oj, 2016.
33. “Ftc ai and impersonation guidance,” https://www.ftc.gov/business-guidance/ ai-tools, 2024.
34. E. Bender et al., “On the dangers of stochastic parrots,” FAccT, 2021.
Related Articles
2026
Artificial Intelligence in Learning and Teaching
2026
Admin Assist: An AI – Driven Configuration and Orchestration for Enterprise Application
2026
Enhancing Blood Group Identification using pigeon inspired optimization: An Innovative Approach
2026
Eco-Genius: Power Up Smart, Power Down Waste
2026
Crowd-Sourced Disaster Response and Rescue Assistant
2026
Unveiling Deepfake Detection Using Vision Transformers: A Survey and Experimental Study
Share Article
Or copy link
https://www.indjcst.com/archives/explicit-content-generation-in-text-to-image-models-training-mechanisms-moderation-failures-and-safety-limits
*Instagram doesn't support direct link sharing from web. Copy the link and share it in your Instagram story or post.