AI skills aren't one-and-done: the mistakes companies make

The most common assumption is that a skill, once written, keeps working. People really do treat skills this way. A July 2026 study of more than 41,000 skills found that 53% of skills people adopt are never edited afterwards. Another study from the same month found that once a quality problem creeps into a skill, it rarely gets fixed in later versions.

Anthropic, which created the skills format, calls skills "living documents" in its guide to building them. The people who use a skill every day notice where it falls short. If nobody collects what they notice and improves the skill, the same flaws keep running for everyone.

A skill that worked twice for the person who wrote it has proven very little. AI tools give a slightly different answer every time, so a good result is partly luck. Anthropic's engineers show this with simple arithmetic: a skill that gets it right three times out of four will get it right three runs in a row only 42% of the time.

A simple skill can be written in minutes, and some guides suggest exactly that. A skill that runs a real company process is a different job. Mapping the process can take hours. The data it needs is usually spread over several systems. And the most valuable part is often knowledge nobody has written down: the exceptions, the conditions, the "we always check this first" that lives in experienced people's heads.

Getting that knowledge out takes time with the people who have it. Skipping that work makes the skill quicker to build, and wrong in exactly the cases that matter.

When a skill gives poor results, the natural reaction is to add more instructions. We see people pile on detail and still get nowhere. The research points the other way. In SkillsBench, a 2026 research benchmark that tested skills on 87 tasks, short and focused skills improved results far more than long, exhaustive ones.

What a struggling skill usually lacks is structure: clear steps, where the data for each step comes from, the decision points, and a check after each step that confirms it was done right. We go through all of these in What makes a good AI skill.

Asking an AI to help write a skill is sensible, and Anthropic recommends it. The mistake is letting the AI write it alone, from general knowledge, with no structure and no testing. In the same SkillsBench study, skills that AI agents wrote for themselves this way did worse than having no skill at all. Skills curated by people raised the average success rate from 34% to 51%.

The skills that work come from a collaboration. The expert knows how the work is really done, the AI helps put it into words, and both follow a structure the result can be checked against.

A skill can get worse without anyone touching it. The AI models underneath are updated regularly, and an update can change how a skill behaves. Anthropic warns that a skill's behaviour can shift from one month to the next even when nobody has edited it. Builders of custom GPTs report the same thing on OpenAI's own forum: same instructions, a new model, and a GPT that stopped doing the one thing it was built for.

Approving a skill once is therefore not enough. It needs testing again whenever the AI changes, and a review schedule that matches how much the company depends on it.

Each of them comes from treating a skill as written instructions. In practice a skill behaves like a small AI agent. It takes inputs, follows steps, makes decisions and produces a result that has to be right every time. Treat a skill like a document and it fails the way untested software fails: quietly, and for everyone at once.

AI engineers learned long ago what keeps an agent reliable: test cases with expected results, checks along the way, automatic scoring of the output, and testing again whenever something changes. The difficulty is that the people who should own skills are rarely AI engineers. Anthropic has observed that most people who write skills are subject matter experts. They know the process, the exceptions and what a good result looks like. What they miss is the engineering practice, explained in a way that makes sense to someone whose job is not AI.

That gap is what we are building Fygment to close. Every skill in Fygment has an owner, a version history and an approval step. Colleagues send feedback and suggested changes straight to the owner, and each change becomes a new, reviewed version. Skills carry their own test cases, a new version has to pass them before it can be approved, and the same tests run in each AI tool your team uses. While someone writes a skill, Fygment points out what it is still missing.

If your team is starting to rely on AI skills, you can try Fygment for free.