Note: This blog post is the fourth in a series on AI and how to make the most of it in your Security Awareness, Culture and Human Risk efforts. This post covers advanced Prompt Engineering and was authored by Dan deBeaubien and Lance Spitzner. You can access the other blog posts below.
- Part 3: Issues, Challenges, and Limitations of AI
- Part 4: Advanced Prompt Engineering
- Part 5: Generating Images
- Part 6: Analyzing Data
What is Generative AI and Prompt Engineering?
Generative AI is the type of AI you will likely use at home or in the workplace. Generative AI, Gen AI for short, can easily create a wide range of content like business cards, project plans, marketing material, newsletters, music, and images. It can answer complex questions and analyze complex content. Interacting with the AI is easy, and it responds similarly to how a human counterpart would. It is this capability that exponentially increases your productivity while saving you time and money. As we covered in Part 2 of this series, the key to Generative AI is knowing how to tell it what you want, a process called Prompt Engineering. Below is the formula covered in the previous blog post.
Context + The Ask + Output Format = Effective Prompt
In Part 2, we then created an example prompt using the above formula. Specifically, we asked Gen AI to create a video script on vishing. This is the prompt we used:
“I am security awareness officer for my company. We are in the financial industry and concerned that we are becoming more and more targeted with vishing attacks. I need your help in creating a script for a short video on vishing. In this video, be sure to explain what vishing is, what makes the attack so dangerous and the top five most common ways you can detect it, and what to do if you fall victim to a vishing attack. Make sure one of the ways to detect includes sense of urgency. Also, be sure to keep the script simple to understand and write it at a 9th grade level. Ensure that the video is no longer than 2 minutes long.”
Let’s drop this prompt into OpenAI’s ChatGPT4 and see what we get. In the screenshot below, we include only the first part of the complete script.
Go ahead and try this prompt for yourself and see what you get. Impressive, isn’t it? But this is just the beginning of what Gen AI can do. Now, let’s assume you love the format, length, and context of the video script, but now you want to create a quiz based on the scrip.
What is Advanced Prompt Engineering?
When using Gen AI solutions like ChatGPT, you can make your prompts far longer and more specific. Remember, you are not dealing with a human. AI is not going to get upset no matter how demanding you are or how many requests you make. In addition, Gen AI remembers context, so once you create the script for your training video, you can do additional follow-up prompts. For example, when I create training material, I often find one of the most time consuming tasks is creating associated quiz questions. So, I could ask a follow-on question like this:
“Create five quiz questions based on this script. For each quiz question, create one correct answer and two distractors. In addition, include an explanation of why the correct answer is correct.”
Here is the first question AI returned:
This is great, right? The question is well written, the distractors are solid. That said, when we generate this type of content, we can be far more specific. For example, let’s imagine that this isn’t the only quiz we are generating and that we want to use this as part of a content generation process. Here are a few “real-world” things we would probably want to consider:
Format: The format is good, but we may want to improve it and make the process repeatable. We want the AI to produce the format we want to send to any Quality Assurance team and ultimately use within our internal processes and systems.
Difficulty: While the questions are “ok,” we gave the AI no guidance about how easy or hard the questions should be.
Content: The AI was also unguided with respect to what types of questions to ask and what content to include or exclude.
Structure: The AI did a good job on the questions and distractors, but we don’t want to assume when it comes to the format and structure of the questions themselves.
Let’s improve the structure by giving ChatGPT some additional details on what makes for a good question by adding this to the prompt:
- Make sure the correct answer and distractors are roughly the same length.
- Ensure that the verb tenses in both the question and the answer choices are consistent. If the question is in the present tense, keep the answers in present tense as well.
- Be clear and specific with your wording. Vague or ambiguous language can lead to confusion.
- If more than one distractor could be considered correct, word the question similar to “Please select the best answer….”
- Use parallel construction for the distractors.
Now we will be much more specific about the content used to create the quiz:
- When generating the quiz please select questions that specifically relate to the primary learning objectives of the video.
- Do not ask questions about company, the author, or instructor of the video.
- The questions should reinforce and assess the concepts of the video that relate to human risk and cybersecurity.
Ok, now the difficulty level of the question. Note that using any kind of subjective measure is unpredictable, so it helps significantly if we also include a basis for the measurement.
“If difficulty for the question is measured on a scale of 1 to 10, with one being ‘easy’ and 10 being ‘hard,’ please generate questions with a difficulty range from 6-9. An ‘easy’ question is one which could be answered correctly by anyone with even a cursory knowledge of the subject, and a ‘hard’ question is one which is difficult even when paying close attention to the video.”
Lastly, the formatting of the output. Here, we are going to take advantage of the fact that Gen AI responds very well to formatting examples. Rather than simply describing the format, we will further reduce the ambiguity of the prompt by using an example.
Let’s give a specific example of the formatting, but for something totally unrelated. This is VERY useful in any process-oriented AI interaction.
Please format the output as follows:
- For each element of each question, use a tag as denoted below.
- Number each question.
- Use a letter for each response.
- Always make the correct answer choice A.
- The feedback should not reference the choice letter, just the correct choice text.
- The feedback should reenforce the answer, but not say anything like “correct or incorrect” because our course tool already does that.
For example, if the quiz were about peanut butter and jelly sandwiches, questions should always be formatted like this:
Question_Number: 1
Question_Text: What is the best method for cleaning the knife while making a sandwich?
Answer_A: Wipe any excess on the edge of the bread.
Answer_B: Lick the knife clean.
Answer_C: Put the knife away dirty.
Correct_Feedback: We should wipe the knife on the bread and then put it in the dishwasher.
Incorrect_Feedback: You want to ensure you store the used knife clean, start by wiping excess peanut butter on the bread.
Let’s combine all these different elements together into a single prompt and see what the results are.
Create five quiz questions based on this script. For each quiz question create one correct answer and two distractors. In addition, include an explanation of why the correct answer is correct.
- Make sure the correct answer and distractors are roughly the same length.
- Ensure that the verb tenses in both the question and the answer choices are consistent. If the question is in the present tense, keep the answers in present tense as well.
- Be clear and specific with your wording. Vague or ambiguous language can lead to confusion.
- If more than one distractor could be considered correct, word the question similar to “Please select the best answer…”
- Use parallel construction for the distractors.
- When generating the quiz, please select questions that specifically relate to the primary learning objectives of the video.
- Do not ask questions about company, author, or instructor of the video.
- The questions should reinforce and assess the concepts of the video that relate to human risk and cybersecurity.
If difficulty for the question is measured on a scale of 1 to 10, with one being ‘easy’ and 10 being ‘hard,’ please generate questions with a difficulty range from 6-9. An ‘easy’ question is one which could be answered correctly by anyone with even a cursory knowledge of the subject and a ‘hard’ question is one which is difficult even when paying close attention to the video.
Please format the output as follows:
- For each element of each question, use a tag as denoted below.
- Number each question.
- Use a letter for each response.
- Always make the correct answer choice A.
- The feedback should not reference the choice letter, just the correct choice text.
- The feedback should reinforce the answer, but not say anything like “correct or incorrect” because our course tool already does that.
For example, if the quiz were about peanut butter and jelly sandwiches, questions should always be formatted like this:
Question_Number: 1
Question_Text: What is the best method for cleaning the knife while making a sandwich?
Answer_A: Wipe any excess on the edge of the bread.
Answer_B: Lick the knife clean.
Answer_C: Put the knife away dirty.
Correct_Feedback: We should wipe the knife on the bread and then put it in the dishwasher.
Incorrect_Feedback: You want to ensure you store the used knife clean, start by wiping excess peanut butter on the bread.
Here is what the AI returned:
Note the changes in the formatting, content, difficulty, and question structure. And this is much more repeatable and can be applied to almost any content. Whenever you write prompts for Gen AI, your prompts should leave little to the discretion of the AI itself, you want to be as specific as possible. Remember, you are dealing with a computer, not a human, so it will not get upset no matter how demanding you are. Another fantastic example of advanced prompt engineering is from Horatiu Petrescu’s talk on creating ChatGPT prompts for mapping BJ Fogg’s Behavior model against specific high-risk behaviors at the SANS Human Risk Summit.
In our next post we will shift gears from content generation to cover how you can use Generative AI to create imagery.
Interested in reducing your organization’s human risk? Check out my course LDR433: Managing Human Risk and sign up for a FREE course preview here.
