Interrogatory LLM

When we need an LLM to perform a complex task, we often need to feed it a
lot of context. Coming up with a design for a new feature requires
descriptions of how we want the feature to appear to the user, guidelines on
how it should be implemented, information on external systems to consult, and
so on. All this can be several pages of markdown. The obvious way to do
this is for a human to write this context, but an alternative is to use an LLM to write
this context after interviewing a human.

The way I can do this is to prompt the LLM to interrogate me. It should ask me
all the questions it needs to create this appropriate context. I can feed much
of the information it needs, and tell it other sources it needs to consult if
it can’t figure those out itself. Once it’s done, it can then create the
context report for another session (perhaps with another model) to carry out
the next step.

I first saw a decent description of this approach in Harper
Reed’s blog. A striking element of his approach is insisting that the LLM
ask only one question at a time. (When I tried it, I found it needed to be
frequently reminded of this.)

Another way to use an interrogatory LLM is to give it a document, such as a
software specification, that captures knowledge about a domain – and then ask
the LLM to interview a human expert to determine if the document is accurate.
This is an alternative to getting the human expert to read the document to
review it. People often find reviewing hard, so a conversation with an LLM
might be more fruitful, particularly if the document isn’t well-written.

Naturally we can use both of these, using one interrogatory LLM to build a
document, then using other interrogatory LLMs to review it with other experts.

The above is getting an LLM to create or assess context for a particular
use of an LLM. But the technique is more broadly applicable. I’ve become a
natural writer, someone who finds the process of writing an essential part of
thinking. To really understand something, I need to write about it. But
different people are different. Many folks find writing hard, often
very hard. This can be a real problem when we need to get information
out of someone’s head into a form that other humans can consume. Maybe such
people would find it easier to ask an LLM to interview them than to write a
document themselves. Certainly the result will have that tang of AI-writing
that folks like me shudder at – but that’s better than not having the
information itself, either due to rushed writing or no writing at all.