Domain oriented Metacognition - the missing skill for reason based prompt engineering with OpenAI o1

I said last week that OpenAI o1 - Reasoning (and not data) is now the new Oil

I believe that reason based prompting will be a key skill for the future in the sense that you will develop long, thoughtful prompts to engage with advanced AI. That's different from our current prompting approaches for most cases. But the next question is .. how do we develop and teach this ability for reason based prompting?

Here is a thought I am exploring (with the aid of #openAI O1 preview of course)

The idea of combining metacognition with reasoning for prompting (with respect to OpenAI o1),

Metacognition, of course is not new. Essentially, metacognition is 'thinking about thinking' i.e. a reflective process.

So, to apply metacognition to reason based prompting, you could approach this in three ways:

1. Before Prompting:

2. During Interaction:

3. After Response:

In each case, you would develop a Metacognitive Questioning Strategy

Here are sample questions for each of these three stages.

This approach of developing a Metacognitive Questioning Strategy helps to create a concrete actionable strategy for implementing reasoning in prompt engineering. I believe that domain oriented metacognition for reason based prompts is a potential new skill for the future.

1. Before Prompting

Purpose: To plan and clarify your objectives before interacting with the AI.

1. What specific information am I seeking from the AI?
2. Do I fully understand the topic, or are there areas I need to research first?
3. What is the main goal of my prompt?
4. How can I phrase my question to be as clear and precise as possible?
5. Are there any assumptions I'm making that should be stated explicitly?
6. What keywords or phrases should I include to guide the AI effectively?
7. Should I provide context or background information to enhance the AI's response?
8. Am I asking an open-ended question that encourages detailed reasoning?
9. How might the AI interpret my prompt—could it be misunderstood?
10. What examples or specifics can I include to clarify my request?

2. During Interaction

Purpose: To monitor and adjust your approach in real-time as you engage with the AI.

1. Is the AI's response aligning with my expectations so far?
2. Am I receiving the depth and detail of information I need?
3. Are there any misunderstandings evident in the AI's response?
4. Do I need to provide additional clarification or rephrase my prompt?
5. Is the conversation staying on topic, or is it veering off course?
6. Am I asking follow-up questions that delve deeper into the subject?
7. Is there jargon or terminology that needs to be simplified or defined?
8. Should I break down my question into smaller parts for clarity?
9. Am I maintaining an open-minded approach to new information presented?
10. Is the tone and style of the AI's response appropriate for my needs?

3. After Response

Purpose: To reflect on the effectiveness of your prompt and the quality of the AI's response.

1. Did the AI fully address my question, or are there gaps in the information provided?
2. What aspects of my prompt led to a successful or less effective response?
3. How could I improve my prompt to get a better answer next time?
4. Did I learn something new or gain a deeper understanding of the topic?
5. Are there any biases or inaccuracies in the AI's response that I need to consider?
6. What follow-up questions can I ask to expand on this information?
7. Did the AI provide reasoning and evidence to support its answer?
8. How well did I use metacognitive strategies during this interaction?
9. What patterns am I noticing in how the AI responds to different types of prompts?
10. How can I apply what I've learned from this interaction to future prompts?

If you want to study with us , please see our #universityofoxford course on #AI #generativeAI and #mlops . We are also applying these ideas to the erdos community