BytePlainlab
Discuss Tech 交流技术
AISafetyType SystemsPlain Language

AI-written code always has bugs nobody has found yet. Here is how I use it anyway. AI 写的代码总有没找到的 bug,我照样大量用它,靠的是闸门

I use AI heavily, and every time I ask it to review its own code, it finds new bugs. That is worth sitting with. AI works by probability, it writes a lot of code very fast, and some software runs things in the physical world, where one bug can put a person at risk. Not using AI means falling far behind. Using it means more unfound bugs on every pass. This note explains, in plain terms, how I weigh the two. 我大量用 AI,而每次让它审它自己写的代码,它总能再找出新 bug。这件事值得停下来想一想:AI 靠概率工作,写代码又多又快,而有些软件控制的是物理世界里的东西,一个 bug 就可能伤到人。不用 AI,效率会被远远甩开;用它,每一轮都多出一批没找到的 bug。这篇用大白话讲讲我怎么权衡。

2026-08-03 · updated 2026-10-09 · first published on boheastill.com最初发表于 boheastill.com

Why the bugs never run out

I fix the bugs I know about. The ones I don’t know about, I have no way of knowing. So I will never finish fixing bugs, and neither will AI: each review just finds a different unfinished batch. That means safety can’t depend on anyone’s vigilance, mine or a model’s.

Sort mistakes by what they cost

Instead of trying to find every bug, list what can go wrong when software runs a machine, from mild to severe:

  • Something on screen looks wrong: annoying, harmless.
  • The machine won’t move.
  • The machine damages itself.
  • A person gets hurt.
  • A catastrophic loss.

There are only a handful of these classes, and they don’t grow as the code grows. That makes them something you can actually check.

Gates, like a type system

For each dangerous class, build a hard check that every command must pass before it reaches the machine. If you have never used a typed language such as Java, Go or Rust: its compiler refuses to run a program when a value is the wrong kind, the way a socket won’t take the wrong plug, however careful the programmer was that day. A gate works the same way. If the preconditions for a command aren’t met, the command doesn’t go through.

The software will be wrong someday. The gates’ job is to make sure being wrong can’t turn into harm.

Where AI fits

Once the gates sit in a layer that neither I nor the model can write or bypass, AI’s speed becomes safe to use. I use it heavily, on the non-core parts, after I have made the decisions. The division of labor is explicit: AI generates, I decide, the gates enforce. Safety no longer depends on how careful anyone happened to be on a given Tuesday.

The same idea works away from machines. An AI agent that can move money should only propose, while an authorization layer it can’t touch does the executing: Governing AI agents that touch money. An assistant that holds private data should have exactly one way out, and that way out should be checked: A private AI assistant and its outbound boundary.

Written from my work on the operator interface for a force-controlled industrial robot (a client integration project, not the robot maker’s own software), with all client, vendor and product details removed. And in keeping with the subject: AI helped me write this page. The thinking is mine.

Related: the full method behind the consequence classes, Industrial reliability: count consequences, not bugs (on boheastill.com). The wider argument: Where value goes when building is free.

为什么 bug 永远修不完

我修 bug,是因为知道它们在那里。不知道的,我也没办法知道。所以 bug 我永远修不完,AI 也一样,每审一轮,它只是找到另一批修不完的。这说明,安全不能靠任何人的警惕,不管是我的还是模型的。

按代价给错误分类

与其想把 bug 找全,不如把软件控制机器时可能出的错列出来,从轻到重:

  • 屏幕上显示不对:烦人,但无害。
  • 机器不动了。
  • 机器把自己弄坏了。
  • 伤到人。
  • 灾难性损失。

这样的类别只有寥寥几种,也不会随着代码变多而增加,所以是真正能逐一检查的东西。

闸门,像类型系统那样

给每一类危险的错建一道硬性检查,每条指令到达机器之前都必须通过。如果你没用过 Java、Go、Rust 这类强类型语言:它们的编译器发现值的类型不对,就拒绝运行程序,就像插座不接错的插头,不管程序员那天有多仔细。闸门也是这样,一条指令的前提条件不满足,它就过不去。

软件总有出错的一天。闸门要做的,是保证出错不会变成伤害。

AI 放在哪里

闸门一旦放在我和模型都写不到、也绕不过的那一层,AI 的速度就可以放心用了。我大量用它,用在非核心的部分,并且是在我做完决定之后。分工很明确:AI 负责生成,我负责决定,闸门负责执行。安全不再取决于某个周二谁碰巧有多仔细。

同样的思路离开机器也成立。能动钱的 AI Agent 只能提议,真正执行的是一层它碰不到的授权,见让碰钱的 AI Agent 受治理。手里有私密数据的助手,对外只能有一个出口,而且这个出口要被检查,见私有 AI 助手的出站边界。

根据我为一台力控工业机器人做操作员界面的真实集成项目整理(客户项目,不是机器人厂自己的软件),客户、供应商和产品细节均已删除。另外,和本文的主题一致:这一页是 AI 帮我写的,想法是我的。

相关:分类背后的完整方法,见工业可靠性:换掉分母(在 boheastill.com)。更大的论证:当“造东西”变免费,价值去了哪。