I wanna talk about doing really hard tasks with AI, those tasks where the quality of the output is really important. And there's a lot of different priorities that the AI needs to consider and keep in mind.
At our SEO agency, these are things like coming up with strategy and writing content. It's one thing to articulate everything that's important to keep in mind while going through these tasks. But I wanna talk about a pattern that's becoming very popular recently for making sure the AI sticks to those standards.
I've seen a couple different names for this pattern, but I'm gonna call it iterative self refinement. The idea is that you present the AI a rubric, or a set of heuristics by which it can evaluate its own output to determine if it's hitting the mark.
So the model is given a task, let's say, creating a blog post. And after following all that guidance to create the blog post, it's then given this rubric. These would be things like, how well does the article follow our SEO writing guidelines?
How well does the content match the brand voice? How well does the content satisfy the intent of someone who's searching for our target keyword?
The model then evaluates its own output on each of these metrics. Then wherever the output is falling short of a perfect 10 out of 10, the model then goes back and makes improvements until all metrics are a solid 10 out of ten.
This can be very powerful if structured the right way, but there are a few pitfalls to watch out for. I recommend structuring the process using a Claude skill for complex processes. You might even use multiple Claude skills with an orchestrator skill.
If you're not sure, just explain the entire process to Claude and then ask it how it thinks you should organize the skills.
Skill should walk Claude or whatever model you're using through the entire process of generating the first version of the output, along with any guidance it would need to do the job well. The iterative self refinement is just the last step of the process. However, because it should be looping through it several times, it's where it's going to spend the majority of its time on the task.
The key to getting this to work well is creating a rubric that is rigorous but not rigid. You want to take your time in defining clearly and meticulously what success looks like without dictating arbitrary rigid guidelines like specific word counts or keyword density.
If you're lazy with this step and you just provide vague or high level goals, what you're gonna find is the models will typically rate themselves very favourably. They might go through one round of revisions and not make any meaningful edits.
On the other hand, if your guidelines are too rigid, the output's gonna be too robotic and deterministic, and you're missing out on most of the intelligence of these models.
When done right, the model should loop several times. 7 that I recommend, this could become very expensive very quickly. If the task isn't worth that expense, you might consider giving it a maximum number of revisions, or doing the iterative self refinement step in a separate chat.
But this last step is where the real magic happens. After you've gone through the whole process and you have the final output, tell Claude to take everything it's learned through that process and update the skill.
