Strengthen confidence in your templated prompts
1️⃣ Design prompt templates in an extended playground
2️⃣ Define test-queries with expected result json schema or value
3️⃣ Test prompts on entire query collections at once
4️⃣ Schedule automated test runs
➕ more
Hi PH 👋!
Ram here, the creator of Promptotype- a development and testing platform for LLM tasks.
Promptotype is not very new, but it’s the first time it’s launched it here. I started building it while working on another product that used LLM tasks, and realized there are many challenges around this use-case- challenges that are quite easy to solve with the right platform. I also developed a belief in the strength of the LLM tasks use-case.
What are LLM structured tasks?
On a very high level, they’re tasks done by an LLM that can be integrated into our program’s flow- basically a function that’s written in (a somewhat) natural language instead of code. Classic examples can be Entity Extraction, or Sentiment Analysis.
The tasks are typically deterministic- having one correct output for every input. They’re also potentially complex enough to return multiple values, therefore requiring a structured output (as a JSON for example), so our program will be able to work with it.
These qualities require specific prompt-engineering techniques, but also enable easier testing.
Promptotype provides you with a playground for the development, testing, automation, and fine-tuning for the specific use-case of LLM structured tasks.
Features:
🎨 Design your prompt templates in an extended playground.
📋 Define test-queries with expected result json schemas or values.
It's automatic! Run your query through the model to auto-fill the expected result.
▶️ Test prompt templates on entire query collections at once.
Create basic collections for development, extended ones to verify production readiness, and monitor reliability with scheduled automated runs.
📚 Manage your prompts and models
🗄️ Follow up with the entire history of your runs and tests.
⏰ Schedule periodic automated tests.
Get result summary by email immediately.
🧮 Everything works with function calling too!
🔧 UI for fine-tuning.
Automatically create a fine-tuned model from your query collections.
You’re all welcome to try it out! And if you haven’t thought about integrating LLM tasks to your system, I really recommend checking it out- super powerful use-case imho.
Thanks,
Ram
Plan Quest (Beta)