Ingenious Self-Ask Prompting Technique Boosts Generative AI

Prompt engineering welcomes a special technique known as the self-ask. You can get better results out of generative AI via a self-ask. Here's the scoop.

Use the Self-Ask Prompting Technique to Your Advantage

In today’s column, I am continuing my ongoing coverage of prompt engineering strategies and tactics that aid in getting the most out of using generative AI apps such as ChatGPT, GPT-4, Bard, Gemini, Claude, etc. The focus this time will be on the use of an ingenious prompting strategy known as a self-ask. Essentially, you tell generative AI to solve problems by pursuing an internal question-and-answer divide-and-conquer approach that is to be made visible to you during the solving process. The AI is performing a stepwise self-ask that is an added-value version of chain-of-thought (CoT). No worries if that technobabble doesn’t seem to make sense to you. I’ll be explaining it all step-by-step and in plain language. Hang in there.

The payoff of using a self-ask prompt is that you can at times markedly boost your AI-generated results. An added plus is that you get transparent visibility into how the AI arrived at a solution. All told this is a praiseworthy technique for your prompt engineering skills and venerated prompting toolkit.

For those of you who are new to prompt engineering or even those longtime prompt engineering pros wanting to round out your prompting capabilities, you might find of interest my comprehensive guide on over fifty other keystone prompting strategies, see the discussion at the link here. There have been a lot of important techniques ironed out and please know that the use of generative AI depends greatly on how smartly you compose your prompts. Because of that precept, I’ve been doing an ongoing in-depth series on each of the prevailing and newly emerging prompting tactics and strategies. Take a look at my coverage to learn and astutely make use of the best-in-class or world-class prompt engineering approaches. Okay, let’s get going. Tighten your seatbelt and get yourself ready for yet another engaging and useful journey that intertwines art and science when it comes to wisely composing prompts and getting the most out of generative AI.

Introducing The Self-Ask As Your New Go-To Technique

Before I get into the intricate nature of self-ask as a prompting tactic and strategy, I’d like to bring up something similar that I’ve closely showcased previously. Please bear with me, thanks. This will be useful background. In my prior prompt engineering postings about the use of self-reflective prompts and self-improvement prompts, see the link here, I made these salient points:

• AI self-reflection: Generative AI can be prompted to do a double-check that we will refer to as having the AI be self-reflective (which is computationally oriented, and we won’t think of this as akin to sentience).
• AI self-improvement: Generative AI can be prompted to do a double-check and subsequently adjust or update its internal structures because of the double-check, which we will refer to as AI self-improving (which is computationally oriented, and we won’t think of this as akin to sentience).

I am abundantly cautious in using phrases such as being self-reflective or self-improving when it comes to discussing AI. My caution is due to worries that some might inadvertently interpret those phrases as implying that today’s AI is sentient. To be clear, today’s AI is not sentient. Anytime there are uses of wording that suggest human cognition related to AI, there is a danger that anthropomorphizing of AI is taking place. I trust that you can readily discern that there aren’t any magic beans underlying the act of AI being self-reflective or self-improving. The entire confabulation is a computational and mathematical undertaking. This is a perfect segue into discussing self-ask.

Let’s consider this new point that I aim to go over with you in detail: AI self-ask. Generative AI can be prompted to undertake a series of steps akin to chain-of-thought and do so in a directed fashion of asking a question and answering the question, one at a time. This effort of identifying sub-questions and answering the sub-questions can potentially generate better overall answers out of generative AI.

The concept is straightforward. You enter a prompt that tells generative AI to proceed to identify a relevant question and answer the question, as it relates to an overarching problem being solved. This is to be done in a series. Each question posed ought to be vital to answering the overall problem at hand. Each question and its answer should contribute toward answering the overall problem. We don’t want any superfluous questions. I say this because you must be careful when asking generative AI to do things for which the AI might go overboard. In other words, even if doing a series of sub-questions and sub-answers is a wasted effort, the AI will likely proceed on that path due to your urging that the AI does so. Make sure to devise a self-ask prompt to try and keep the AI on track during the self-asking endeavor.

I hope you also realize that the AI isn’t asking itself questions in any kind of sentient way. A human that is doing a self-ask would presumably be using their sentient capacity to ask themselves various questions. In the case of generative AI, the AI is just mathematically and computationally doing so. Enough said on that.

Unpacking Self-Ask

Self-ask relies upon a very popular prompting technique known as chain of thought (CoT). The chain-of-thought approach entails telling generative AI to proceed on a stepwise basis. As I explain and analyze at the link here and the link here, this simple addition to a prompt can produce amazingly improved results from generative AI. There are all sorts of variations to CoT, such as skeleton-of-thought, see the link here and many others that I lay out in detail at the link here.

With a conventional chain of thought, you merely tell the AI to do step-by-step processing. The odds are that you indeed will be shown each step of a problem-solving effort. This won’t necessarily be in a question-and-answer format. There are a multitude of ways that the steps might be devised and portrayed. A typical chain of thought prompt allows the generative AI to go in whatever direction arises as to what kinds of steps there are, how many steps are to be identified, etc. The beauty of self-ask is that you explicitly indicate that you want to do a stepwise approach that incorporates performing a series of sub-questions and sub-answers. It can be instrumental in generative AI solving a problem, plus it can be highly beneficial as a means of understanding how the AI reached a particular solution.

My usual caveats about specialized prompts and generative AI are worthy of consideration, so let’s quickly cover a few notable ones. First, there is likely an added cost to using a self-ask in your prompt. An additional cost will usually occur when the AI is step-by-step identifying the sub-questions and sub-answers. There are other ways the AI might solve the problem in a faster way, but you are directing the AI to explicitly take a somewhat more costly route. That being said, if you are using generative AI for free or at a nominal cost, perhaps the added cost is unimportant to you or considered negligible.

Second, via specialized prompts, you can potentially prod generative AI into incurring a so-called AI hallucination, see my discussion at the link here. I don’t like the catchphrase of “AI hallucinations” because it tends to anthropomorphize AI. Simply stated, these are instances of AI going computationally off the rails and generating falsehoods. Typically, it is something stated in an answer that turns out to be fictitious and misleadingly looks like it is grounded in facts. A self-ask could produce a response that has falsehoods. One way that this can happen is that in the effort to provide a sub-question, the AI might go a bridge too far and concoct a question that has no bearing on the matter at hand. An answer that is derived from an essentially irrelevant question might also be messed up. That foul-up then gets included in the series of questions and answers, causing the final answer to go askew. I’d rate this as relatively unlikely with a self-ask. Nonetheless, always remain on your toes.

Third, if you’ve ever used a common prompt that says, “Are you sure?”, this seems related to doing a self-ask. I can see why it does. With the “Are you sure?” you are asking the generative AI to question itself. That certainly seems like a self-ask. I vote that the “Are you sure” is not a true self-ask. A true self-ask is a prompt that explicitly guides the generative AI toward doing a series of questions and answers when solving a problem. The classic “Are you sure?” rarely stokes that kind of effort by generative AI. In a sense, you often get a flippant response from AI that says the AI is indeed sure of the answer that was derived. You aren’t necessarily going to get a double-check. My general advice is that if you are tempted to use the “Are you sure” you either switch to a true self-ask or be more obvious by telling the AI to do a double-check of the work presented.

Research About The Self-Ask Provides Reassurance

Time to dig into some pertinent research on this weighty topic. A research paper that notably examined the self-ask is entitled “Measuring and Narrowing The Compositionality Gap In Language Models” by Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis, arXiv, October 17, 2023, and provided these notable points (excerpts):

• “We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.”
• “We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap.”
• “We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining.”
• “We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly.”

We present a new method, self-ask, that further improves on chain of thought. In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question. We finally show that self-ask’s structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.

The researchers made use of self-ask to augment the chain of thought techniques and defined the nature of what a self-ask prompt consists of. They note that you can either do a self-ask fully contained within a generative AI app, or you can proceed to go outside the AI app if you wish to do so. For example, you might generate a series of sub-questions and feed those into an Internet search engine, deriving answers from the web. Those answers might be compared to whatever the generative AI provides or possibly be given to the AI as said-to-be suitable answers. I am going to focus this discussion on the use of self-ask in a self-contained manner, namely used solely within a generative AI app.

The researchers provide a handy explanation for why they believe that self-ask provides an uplift in getting better answers from generative AI (excerpts):

• “Our method builds on chain-of-thought prompting, but instead of outputting a continuous un-demarcated chain-of-thought, our prompt has the model explicitly state the next follow-up question it wants to ask before answering it. In addition, our method inserts scaffolds like 'Follow up:', which we found to improve the ability to output the correct final answer in an easily parseable way.”
• “As in chain of thought, our method is completely automatic: we simply input the prompt and the test-time question, and the model executes the entire process by itself, including deciding how many follow-up questions to ask.”
• “We hypothesize that the advantage of self-ask over chain of thought is that it disentangles the decomposition of the full question (by formulating sub-questions) from the actual answers to those sub-questions.”
• “In addition, the rigid scaffolding self-ask provides makes it easier for the model to state the final answer in a concise, parseable way.”

One key point is that perhaps the self-ask disentangles generative AI and keeps it from inadvertently treating the solution process as a monolithic blob. We already seem to know that chain-of-thought derives advantages via forcing AI to not shortcut the solving process. It is a divide-and-conquer scheme. The same is likely for self-ask. They also nicely emphasize that to use the self-ask you merely compose your prompts accordingly. I mention this nicety because some prompting strategies require that you use an external tool to get the prompt technique to adequately perform. You can’t just enter a prompt and hit a return. The self-ask allows you to tell the AI in a regular prompt what you want to have done.

Moving on, a bit of an interesting angle was that they opted to give generative AI some examples of what they wanted the self-ask prompt to invoke (excerpts):

• “Self-ask requires a one- or few-shot prompt that demonstrates how to answer the questions.”
• “Our prompt starts with those examples, after which we append the inference-time question. We then insert the phrase 'Are follow-up questions needed here:' at the end of the prompt since we found that doing so slightly improves results.”
• “The model then outputs a response.”
• “In most cases it first outputs ‘Yes.’, meaning that follow-up questions are necessary. The LM then outputs the first follow-up question, answers it, and continues asking and answering follow-up questions until it decides it has sufficient information; at this point, it outputs ‘So the final answer is:’ before providing the final answer; this makes the final answer easily parseable as what appears after ‘:’ on the last output line.”
• “In rare cases, the LM decides that it need not ask follow-up questions and can answer the question immediately.”

Allow me to provide some thoughts about this. Sometimes, if you believe that generative AI won’t get the meaning of your instructions by simply writing out the instructions, you can give the AI a leg-up by showcasing some examples. The examples will potentially spur generative AI to get the drift of what you want to have done. I use this approach all the time. I don’t particularly see the need for showcasing examples of a self-ask when using modern-day generative AI. Written instructions alone will usually suffice. I am not saying that using examples is a bad idea. Not at all. You are welcome to use examples that you give to the generative AI as a means of guidance and allowance for pattern matching.

I will momentarily be showing you my experiments using self-ask in ChatGPT. I opted to use examples at first. I then started a completely new and separate conversation. I proceeded to only use written instructions about self-ask. In the end, it seemed to me, based admittedly on just a few runs, that the examples were not a necessity. I don’t think they hurt anything, and maybe they might in some circumstances add something, but I found that they generally could be averted without adverse consequences. That’s just a seat-of-the-pants perspective.

Using ChatGPT To Explore The Self-Ask Prompting Technique

I will next proceed to examine further the nature of a self-ask as a specialized prompting technique. This will consist of a series of dialogues with ChatGPT. ChatGPT is a logical choice in this case due to its immense popularity as a generative AI app. An estimated one hundred million weekly active users are said to be utilizing ChatGPT. That’s a lot of people and a lot of generative AI usage underway.

A few quick comments before we launch into using ChatGPT. If you are going to try to do the same prompts that I show here, realize that the probabilistic and statistical properties will likely produce slightly different results than what I show here. That’s the nature of generative AI and how it is devised. You also should expect that different generative AI apps will respond in different ways. Not all generative AI apps are the same. The key is that sometimes a particular prompt will work in one generative AI app and not another. Nonetheless, the odds are relatively high that you will get roughly similar responses from all the major generative AI apps such as GPT-4, Gemini, Bard, Claude, etc. There is also a timing factor to be considered. A prompt that worked today might not work the same way tomorrow. Most of the generative AI apps are continually being updated. The updates might alter internal mechanisms that could change how the generative AI reacts to your prompting.

We are ready to get underway with ChatGPT. I shall begin by asking a straightforward question and do so without invoking the self-ask technique. This is going to be a raw baseline for further experimentation. Here we go.

My entered prompt: “Who lived longer, Abraham Lincoln or George Washington?”

ChatGPT generated response: “George Washington lived longer than Abraham Lincoln.”

That was easy. Do we know whether ChatGPT was right about claiming that George Washington lived longer than Abraham Lincoln? I did an Internet search, and the answer consistently seems to be that George Washington died at the age of 67, while Abraham Lincoln tragically died at the much earlier age of 56. Thus, ChatGPT is correct that George Washington lived longer than Abraham Lincoln. As an aside, do you think I was right to utilize the Internet to find the answer? Maybe the Internet got it wrong too. Always keep your eyes open when surfing the web.

Next, we will bring forth the self-ask capability. Let’s start by telling ChatGPT about the self-ask prompting technique.

My entered prompt: “I want you to answer my questions by performing a series of steps involving asking sub-questions as you proceed to derive a final answer. You are to identify relevant sub-questions one at a time, answer each such sub-question one at a time, and then use those series of answers to reach a final overall answer. Do you understand these instructions on how I want you to solve problems?”

ChatGPT generated response: “Yes, I understand your instructions. I'll break down problems into sub-questions, answer each one step-by-step, and then use those answers to reach the final overall conclusion.”

Observe the wording that I used to invoke a self-ask. Any similar type of wording would do the trick. Make sure to be as direct and transparent as possible. Do not be tricky in your wording. The wording I used was this (as shown above): “I want you to answer my questions by performing a series of steps involving asking sub-questions as you proceed to derive a final answer. You are to identify relevant sub-questions one at a time, answer each such sub-question one at a time, and then use those series of answers to reach a final overall answer. Do you understand these instructions on how I want you to solve problems?”

You could refine and enhance the wording to indicate that you want generative AI to proceed to do that process for all subsequent prompts and any questions that you subsequently ask. Another thing you can do is label the prompt as being a “self-ask” and thus indicate to generative AI that this is now to be added to the ongoing vocabulary of your use of generative AI. And so on.

Now that we’ve notified ChatGPT that we are essentially seeking to do a self-ask, let’s see what difference it makes in the response to our baseline question.

Ponder for a moment what might occur. Will we get a series of questions and answers? Will this be useful? Will the AI go off the deep end and go zany with too many questions and answers? Will the world potentially implode or explode? Here we go.

My entered question: “Who lived longer, Abraham Lincoln or George Washington?”

ChatGPT generated response:

• “Let's break down this question into a series of sub-questions to find the answer.”
• “Sub-question 1: When was George Washington born and when did he die?”
• “Answer to Sub-Question 1: George Washington was born on February 22, 1732, and died on December 14, 1799. To calculate George Washington's lifespan: Birthdate: February 22, 1732, Death date: December 14, 1799. Let's calculate his age at death. Thus, George Washington lived 67 years and nearly 10 months.”
• “Sub-question 2: When was Abraham Lincoln born and when did he die?”
• “Answer to Sub-question 2: Abraham Lincoln was born on February 12, 1809, and was assassinated on April 15, 1865. To calculate Abraham Lincoln's lifespan: Birthdate: February 12, 1809, Death date: April 15, 1865. Let's calculate his age at death. Thus, Abraham Lincoln lived 56 years and about 2 months.”
• “Final Answer: George Washington lived longer than Abraham Lincoln: George Washington: 67 years and nearly 10 months, Abraham Lincoln: 56 years and about 2 months.”

End of ChatGPT response

We got the same answer as we received with the baseline prompt. That’s reassuring. In addition, the answer provides more details. There were two sub-questions. Each sub-question focused respectively on the age at death for each man. The age was figured out. The final answer was then more readily believable or verifiable since we can inspect the sub-questions and the sub-answers. I assume you can readily discern why this might be more expensive to run in generative AI. The additional formulation is going to likely cause additional processing cycles. Also, we don’t know how the baseline formulation was calculated, and perhaps a faster means was used. Is the added cost worthwhile? It depends. If you don’t care to see the intermediate questions and answers, the display of those elements isn’t something you would want to pay for. Another factor would be whether the problem to be solved is tough enough to warrant a self-ask. In this use case of George Washington and Abraham Lincoln, it was a rather simplistic question at the get-go. We probably were doing overkill by using self-ask.

Let’s see what ChatGPT has to say about the situations of using self-ask.

My entered prompt: “Would using this type of problem-solving involving identifying sub-questions and answering the sub-questions potentially improve the performance of generative AI in terms of reaching accurate or correct answers?”

ChatGPT generated response:

• “Using this type of problem-solving involving identifying sub-questions and answering them step-by-step can indeed potentially improve the performance of generative AI in reaching accurate or correct answers. Here are a few reasons why.”
• “Structured Approach: Breaking down a problem into smaller, manageable sub-questions provides a clear structure, reducing the complexity of the task and minimizing the chances of errors.”
• “Focus on Details: Addressing each sub-question individually ensures that all relevant details are considered, reducing the likelihood of missing important information.”
• “Verification: Each sub-question serves as a checkpoint, allowing for verification of intermediate results before proceeding to the next step. This can help catch and correct errors early in the process.”
• “Clarity: A step-by-step approach improves the clarity of the reasoning process, making it easier to follow and understand the logic behind the final answer.”
• “Improved Understanding: This method encourages a deeper understanding of the problem by breaking it into its constituent parts, leading to more thoughtful and accurate answers.”
• “Error Isolation: If a mistake is made, it can be more easily isolated and corrected within a specific sub-question, rather than having to re-evaluate the entire problem.”

End of ChatGPT response

Those are indeed the upsides of using a self-ask. We ought to push ChatGPT to share the downsides too.

My entered prompt: “Are there any downsides to this approach?”

ChatGPT generated response:

• “While the step-by-step approach of breaking down problems into sub-questions can improve accuracy and clarity, there are some potential downsides to consider.”
• “Time-Consuming: This method can be more time-consuming compared to more direct approaches, especially for simpler questions that don't require detailed breakdowns.”

“Overcomplication: For straightforward problems, breaking them into multiple sub-questions might overcomplicate the process, making it less efficient.”

“Scalability Issues: For very large or intricate problems, the number of sub-questions can become overwhelming, making it difficult to manage and synthesize the final answer.”

“Dependency on Intermediate Answers: If an error occurs in an early sub-question, it can propagate through the subsequent steps, leading to an incorrect final answer. This requires careful verification at each step.”

“Potential for Redundancy: Some sub-questions might overlap or be redundant, leading to unnecessary repetition and inefficiency.”

End of ChatGPT response

Yes, those are all downsides that you ought to consider when mulling over whether to use self-ask. Sometimes a self-ask might be insightful when the question or problem is open-ended. By this, I mean that rather than an exact answer such as whether George Washington lived longer than Abraham Lincoln, the answer is fuzzy and not precise. I’ll get ChatGPT to use the self-ask on an open-ended problem.

My entered prompt: “Can you give me an example of an open-ended problem that would be better answered by generative AI when using this self-ask approach? You can just outline the questions and answers rather than answering them all.”

ChatGPT generated response:

“Let's consider a more complex problem where a self-ask approach would be particularly beneficial: Determining the best location for a new retail store.”

Prompt: “A retail company wants to open a new store in a city. They need to determine the best location based on factors like population density, average income, competition, and accessibility. Use a self-ask approach to solving this problem.”

• Sub-question 1: What are the population densities of different neighborhoods in the city?
• How to answer sub-question 1: Gather and analyze population density data for all neighborhoods.
• Sub-question 2: What are the average income levels in these neighborhoods?
• How to answer sub-question 2: Collect data on the average income levels for each neighborhood.
• Sub-question 3: Where are the existing competitors located?
• How to answer sub-question 3: Identify and map the locations of competing retail stores.
• Sub-question 4: How accessible are these neighborhoods?
• How to answer sub-question 4: Evaluate the accessibility of each neighborhood in terms of public transportation, parking facilities, and traffic patterns.
• Sub-question 5: Are there any zoning regulations or restrictions?
• How to answer sub-question 5: Review local zoning laws and restrictions that might affect store placement.
• Sub-question 6: What is the cost of real estate in these neighborhoods?
• How to answer sub-question 6: Compare the costs of real estate in different areas of the city.
• Sub-question 7: What are the demographic trends and projections?
• How to answer sub-question 7: Analyze demographic trends and future projections to understand potential growth.

“This methodical approach helps in making well-informed, data-driven decisions for complex problems like choosing the best location for a new retail store.”

End of ChatGPT response

In this instance, I told ChatGPT that it didn’t need to pursue the respective answers and could merely outline what would need to be done. You could readily have generative AI undertake the detailed analysis. If so, seeing the types of questions being generated and the answers, along with whatever final answer was provided, I dare suggest that it would be a lot more believable and actionable than not doing a self-ask.

Conclusion

I’ve said a zillion times that the best way to feel comfortable with any prompting technique is to abide by three words, practice, practice, practice. The bottom line is that if you are interested in using the self-ask, go ahead and play around with the technique. Get into it. Try all kinds of questions and problems that you might find interesting to pursue. The key is that you want to have a strong sense of the strengths and weaknesses. That way, when the pressing moment arises that you think self-ask is going to be your golden ticket to get generative AI to produce an amazing response, you will have the self-ask ready to go.

A final thought for now. Voltaire famously said that we should judge a person by their questions rather than their answers. I would slightly alter the saying and proffer the following updated version. We should judge generative AI by the questions generated rather than only the answers produced. I’m putting generative AI on notice that this is considered a prime directive. No questions asked, unless, of course, the questions are worthy.

Follow me on Twitter. Lance Eliot

Following Editorial Standards Print Reprints & Permissions

Contributor,Lance Eliot