In nearly all of my recent explorations of Generative AI, I've come to realize how important prompts are. That hasn't necessarily translated me into writing better prompts all the time, but being aware of the problem is at least one step towards correcting it, right? One of the things that I thought would help me personally would be to have a tool to quickly compare and contrast different prompts. Obviously, I could just open up multiple tabs in a browser, but I really wanted something more direct and simplified for my purposes. With that in mind, let me share what I've built.

Prompt Tester

Prompt Tester is my incredibly imaginatively named tool to... um... test prompts. It begins with one prompt:

The idea is that you enter your initial prompt, and then click one or more times to add more prompts. When ready, you can then just click "Generate Results":

Before showing some example results, let me talk a bit about how it was built.

The Code

The backend is an incredibly simple Node.js server that handles throwing up the static HTML and responding to the front end API requests. Once again, I thank Todd Sharp for sharing his initial code with me. Here's the boring simple handling portion:

async function handler(req, res) {
	console.log('Entered handler.', req.method, req.url);

	if(req.method === 'GET' && req.url.indexOf('favicon.ico') === -1) {
		res.writeHead(200, { 'Content-Type':'text/html' });
		res.write(fs.readFileSync('./demo.html'));
		res.end();
	} else if(req.method === 'POST' && req.url === '/api') {

		let body = '';
		req.on('data', chunk => {
			body += chunk.toString();
		});

		req.on('end', async () => {
			body = JSON.parse(body);
			console.log('BODY:\n', JSON.stringify(body, null, '\t'));

			let result = await callGemini(body.prompt);
			res.writeHead(200, { 'Content-Type':'application/json' });
			res.write(JSON.stringify(result));
			res.end();

		});

	}

}

const server = http.createServer(handler);
server.listen(3000);
console.log('Listening on port 3000');

And the callGemini part, which just passes the prompt off to Google's Gemini API. In theory, this could be changed to ChatGPT or any other service.

const MODEL_NAME = "gemini-pro";
const API_KEY = process.env.GOOGLE_AI_KEY;

const genAI = new GoogleGenerativeAI(API_KEY);
const model = genAI.getGenerativeModel({ model: MODEL_NAME });

async function callGemini(text) {

	const generationConfig = {
		temperature: 0.9,
		topK: 1,
		topP: 1,
		maxOutputTokens: 2048,
	};

	const safetySettings = [
		{ category: HarmCategory.HARM_CATEGORY_HARASSMENT, threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, },
		{ category: HarmCategory.HARM_CATEGORY_HATE_SPEECH,	threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, },
		{ category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT, threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, },
		{ category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, },
	];

	const parts = [
    	{text},
  	];

	const result = await model.generateContent({
		contents: [{ role: "user", parts }],
		generationConfig,
		safetySettings,
	});

	try {

		if(result.response.promptFeedback && result.response.promptFeedback.blockReason) {

			return { error: `Blocked for ${result.response.promptFeedback.blockReason}` };
		}
		const response = result.response.candidates[0].content.parts[0].text;
		return { response };
	} catch(e) {
		// better handling
		return {
			error:e.message
		}
	}
	
}

I'm proud to say that I'm now, finally, doing some basic error handling and trying to expose to the user (me) why a prompt was blocked.

The front end is relatively simple. I used Milligram for the core CSS (again, thank you Todd) and some vanilla JavaScript to handle adding new prompts and making the calls. Here's the HTML (minus the inline CSS):

<h1>Prompt Tester</h1>

<p>
This tool helps you see the result of multiple prompts at the same time. The idea is to use similar prompts with different phrasing 
to see how it impacts the result.
</p>

<div id="prompts">
</div>

<template id="promptRow">
<div class="row">
	<div class="column column-33"><textarea placeholder="Enter your prompt."></textarea></div>
	<div class="column column-67 result"></div>
</div>
<hr>
</template>


<div class="float-right">
<button id="addPrompt">Add Prompt</button> <button id="generateResults">Generate Results</button>
</div>

And then the JavaScript. Notice I used <template> above to handle the prompt layout. This is the first time I've used that tag and it was relatively easy.

let $template, $prompts, $addPromptsBtn, $generateResultsBtn;
let promptCount = 0;

document.addEventListener('DOMContentLoaded', init, false);
async function init() {
	
	$addPromptsBtn = document.querySelector('#addPrompt');
	$addPromptsBtn.addEventListener('click',  addPrompt, false);
	$generateResultsBtn = document.querySelector('#generateResults')
	$generateResultsBtn.addEventListener('click',  generateResults, false);
	$template = document.querySelector('#promptRow');
	$prompts = document.querySelector('#prompts');

	addPrompt();
}

function addPrompt() {
	promptCount++;
	const newPrompt = $template.content.cloneNode(true);
	$prompts.append(newPrompt);
}

async function generateResults() {
	// disable add prompt and generate
	$addPromptsBtn.setAttribute('disabled', 'disabled');
	$generateResultsBtn.setAttribute('disabled', 'disabled');

	let $results = document.querySelectorAll('div.result');
	// clear any previous results
	$results.forEach(r => r.innerHTML = '');

	/*
	gather prompts from the inputs
	*/
	let prompts = [];
	let $fields = document.querySelectorAll('#prompts textarea');
	for(f of $fields) prompts.push(f.value);

	let promises = [];
	for(p of prompts) promises.push(getResult(p));
	let results = await Promise.all(promises);

	// assign results to result area for each, handle errors - later
	// there's a one to one correspondance between our array of results and the result div
	for(let i=0;i<results.length;i++) {
		if(results[i].response) $results[i].innerHTML = marked.parse(results[i].response);
		else if(results[i].error) $results[i].innerHTML = `<strong>Error with Prompt: </strong> ${results[i].error}`;
	}

	$addPromptsBtn.removeAttribute('disabled');
	$generateResultsBtn.removeAttribute('disabled');

}

async function getResult(p) {
	console.log('get result for',p);
	// quick return on empty prompt
	if(p.trim() === '') return { response:'<strong>Enter a prompt.</strong>' };
	let req = await fetch('/api', { method:'POST', body: JSON.stringify({prompt:p.trim()})});
	let result = await req.json();
	return result;
}

I think that's all pretty simple DOM manipulation and API calling, but definitely let me know in a comment below if something doesn't make sense. Notice I fire off all my API calls at once and Promise.all wait for them to finish.

Also, note I use the marked library. The results from Google typically have Markdown and this helps render them nicely in my little web app.

All of this isn't terribly interesting I think, but let's look at some testing. As I said, I knew the style of prompt was important. Having it presented directly to me in a tool went a heck of a long way to reinforcing that.

Sample Result - Gravity

Let's start with a simple idea - explaining gravity, but for different age groups. Because of the large size of the results, I'm going to use a bit of CSS here to help keep things sane.

Note the typo in the prompt below. Gemini handled it just fine.

Sample Result - Pets

For my second test, I started with a basic prompt and then a better version that asked for specific types of details:

Sample Result - Chain of Thought Prompting

The following test came from an excellent presentation from AWS re:Invent: AWS re:Invent 2023 - Prompt engineering best practices for LLMs on Amazon Bedrock (AIM377). I found a lot of great advice from this session, including a good overview of Chain of Thought prompting, where the idea is you provide information about the steps used to solve a problem to help the GenAI engine better solve a prompt.

Now, the example given in that presentation went like so:

Q: Roger has 5 tennis balls. he buys 2 more cans of tennis balls. Each 
can has 3 tennis balls. How many tennis balls does he have now?

A: The answer is 11.

Q. The cafeteria had 23 apples. If they used 20 to make lunch and 
bought 6 more, how many apples do they have?

In the presentation, the model in use incorrectly returned 27. In my test, it worked correctly, but I absolutely saw issues with math back when I built my generative AI blackjack advice demo last November.

As I said, Gemini handled it fine: A: The answer is 9. But in Prompt Tester, I tried the suggested CoT prompt:

Q: Roger has 5 tennis balls. he buys 2 more cans of tennis balls. Each 
can has 3 tennis balls. How many tennis balls does he have now?

A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 
tennis balls. 5 + 6 = 11. The answer is 11.

Q. The cafeteria had 23 apples. If they used 20 to make lunch and 
bought 6 more, how many apples do they have?

And the result was improved, showing how it got to the answer:

A: The cafeteria had 23 apples. They used 20 apples for lunch, which 
means they have 3 apples left. They then bought 6 more apples, so 
they now have a total of 3 + 6 = 9 apples.

The answer is 9.

The difference is pretty significant.

The Code

If this seems useful to you at all, feel free to grab the code from here: https://github.com/cfjedimaster/ai-testingzone/tree/main/prompt_tester. As I mentioned, it shouldn't be that difficult to add callChatGPT or some such and use a different service. If you do, let me know!