Mastering Prompt Engineering or the Art of AI whispering

March 05, 2023

Generative AI systems that respond to plain language inputs are a game-changer. It means anyone can interact with AI without needing programming skills. Users can generate complex outputs like code, images, music, or text just by giving clear instructions. This saves time and effort while also unleashing new possibilities for creative expression and problem-solving.

In this post we will explore the following topics:

The importance of prompt engineering
Practical examples of prompt engineering
Resources and tools to get started with prompt engineering

The importance of Prompt Engineering

Anyone who has used generative AI systems knows that getting the best results for a task requires writing very precise instructions. This process is called prompt engineering, and it’s a crucial skill for effective use of generative AI. Crafting effective prompts involves understanding the AI model’s capabilities and limitations, selecting the appropriate input format, and providing precise instructions.

The hottest new programming language is English
— Andrej Karpathy (@karpathy) January 24, 2023

Mastering prompt engineering leads to more accurate and relevant outputs from generative AI systems. Tesla’s former chief of AI Andrej Karpathy describes it as a kind of “large language model (LLM) psychologist”.

Jaon Allen won the Colorado State Fair’s annual art competition last year, he used a combination of multiple Midjourney outputs (AI art generator tool) and manual editing on Photoshop to create “Théâtre D’opéra Spatial”

Théâtre D’opéra Spatial, Jason Allen — Théâtre D’opéra Spatial, by Jason Allen

In this interview ¹, he describes his process and how he spent hours searching for the perfect prompt to create his work.

“I’m not sharing my prompt ever, it’s because the prompt is valuable. I think that it proves there’s skill and time and creativity involved in creating the prompt.” — Jason Allen

Prompt engineering is starting to be recognized as a valuable skill, the superpower to get AI to generate the outputs you want. Companies are starting to hire Prompt Engineers ² and new businesses are emerging around it.

Practical examples of Prompt Engineering

Designing a prompt

A prompt may consist of one or a combination of the following components ³ :

The instruction

A specific task you want the model to perform: Summarize, Recommend, Translate, Explain, Generate, etc.

The context

It allows to add external information or additional context, which guides the model to generate better responses. Example:

Twitter is a social media platform where users can post short messages
called "tweets". Tweets can be positive or negative, and we would like
to be able to classify tweets as positive or negative

Input Data

The input can be the question that we want to answer, examples of answers or an image to edit in the context of image generators.
Output Indicator

Indicates the desired type or format of the output ⁴.

Examples: Format everything in Markdown, output this in a table

Examples of prompting strategies

Few-Shot Prompts

Few-shot prompts are prompts that include a few demonstrations of the task, which improves the performance ⁵.

Example of a Few-Shot prompt (Brown et al.)

Priming Prompt

An advanced technique to interact with an AI Chatbot is to include a priming prompt. This allows you for example to give a personality to the model. For example if you want it to act as an advertiser ⁶ :

I want you to act as an advertiser. You will create a campaign
to promote a product or service of your choice. You will choose
a target audience, develop key messages and slogans, select the
media channels for promotion, and decide on any additional activities
needed to reach your goals.

My first request is "I need help creating an advertising campaign for
a new type of energy drink targeting young adults aged 18-30."

You can find some additional examples here

Chain of thought

This type of prompting tells the model to explain its reasoning before giving the final answer. For instance, using Let's think step by step allows to improve the permormance in reasoning tasks ⁷.

Example of a Zero-Shot-CoT prompt (Kojima et al.)
Chained Prompting

Performing some complex tasks using a single run of a Large Language Model (LLM) can be challenging. To solve this problem, we can chain multiple prompts to achieve better results ⁸.

Story generation with chained prompts (cohere.ai blog)

Image prompting

If you have attempted to generate images with DALL-E, Midjourney, or Stable Diffusion, you may have experienced disappointing results. This is likely due to an incomplete or insufficient prompt. Here are some examples of what your prompt may be missing to achieve fabulous art:

Type: painting, photography, illustration, modelshoot, drawing
Style: futuristic, unreal engine, pixel art, manga graphics, by [artist]
Lighting: natural lighting, cinematic lighting, dramatic lighting
Boosters: awesome, stunning, masterpiece, featured/trending on X
Resolution: 8K, high definition, best quality, insanely detailed

However, creating art with just words can be challenging, but Prompt engineers can now count on ControlNet ⁹ which is a new flexible method for adding an additional input, such as a sketch or human pose, to guide the generation of new samples in AI models.

Resources & Tools

Prompt-Engineering-Guide: This Github repository is a gold mine for AI whisperers, it contains a large list of courses, papers, datasets and tools to get you started with prompt engineering.

Best practices for prompt engineering by Open AI.

learnprompting.org: A free, open source and user friendly manual for prompt engineering.

AIPRM: AIPRM is a prompt template collection for ChatGPT, you can choose your task (SEO, SaaS, Marketing, Art, Programming..) and access a curated list of prompts. I would like to thank my colleague Léo for sharing this.

PromptHero: One of the largest prompt libraries for AI generated art.

Conclusion

Generative AI has enormous potential, and as the tools and models improve, they will become more user-friendly and effective, unlocking fresh opportunities for prompt engineering while potentially streamlining the process.

Prompt engineering will evolve alongside AI tools, leading to a new breed of prompt engineers who specialize in fields like advertising, law, and art, as well as specialized tools. As the practice becomes more widespread, it will transform existing jobs and empower professionals to achieve more.