Evaluation
- Define success before generating.
- Create typical and edge cases.
- Record prompts, inputs, product/model/version and settings where known.
- Run consistently; score against evidence.
- Investigate failures; revise; retest to catch regressions.
- Document limitations and human review.
Criteria: accuracy, instruction adherence, completeness, relevance, clarity, safety, format, source quality, uncertainty. Failure modes include hallucinations, ignored constraints, bias, unsafe output, data leakage, formatting errors, and version changes. Automated evaluation can miss issues.
Application engineering
- API
- Programmatic interface.
- Structured output
- Format constraints; syntax does not guarantee correct values.
- Tool calling
- Model requests a tool; application validates and authorizes.
- RAG
- Retrieves external data; freshness, access, and source quality matter.
- Workflow
- Ordered tasks/checks.
- Agent
- Iterative tool/step selection; autonomy increases safeguard needs.
- Observability
- Logs/traces/metrics; evaluation and monitoring check behavior over time.
Use the simplest reliable system; prompting may be enough.
Generative media
Define deliverable/audience/format/duration; choose platform/model; prompt for subject, scene, style, composition, motion, constraints; use rights-cleared references; iterate and log; inspect/export and check provenance, quality, terms.
A model generates/transforms; a platform provides access/interface/workflow and may expose multiple models. OpenArt is a multi-model creative platform, not one model. Compare control, consistency, editing, export, privacy, licensing, availability. Respect rights/consent; disclose synthetic media where context requires; avoid deceptive impersonation.
Directing AI video
Think in shots, not just descriptions. For each shot, specify the subject, action, setting, shot size and angle, camera/lens intent, movement, lighting, atmosphere, performance and continuity constraints. Plan what changes over time and what must remain stable. A concise shot prompt can be more controllable than an overloaded scene description.
- Write a one-sentence scene objective and emotional beat.
- Storyboard the action into distinct shots with a purpose for each.
- Generate a reference frame or visual bible to guide continuity.
- Generate and evaluate each shot; log model, prompt, settings and failure.
- Revise one variable at a time, then assemble and assess pacing and sound.
See the AI filmmaking learning resources for optional external courses on directing and full production workflows.
Practice
Sketch a tour companion retrieving official information and displaying sources/dates, with human confirmation before action. Create a fictional concert image/video/audio concept and document iterations. For video, write a three-shot sequence, generate each shot, and evaluate framing, movement, continuity and emotional intent.
Agent reliability and alignment checks
When an AI system can use tools or take actions, evaluate not only the answer but also its behavior. A useful assessment framework separates three failure modes. This framework is a practical teaching lens; do not treat a leaderboard or index score as proof of safety.
- Unauthorized action
- The system takes an action beyond what the user requested or approved. For example, it sends a message when asked only to draft one, or changes a file without approval.
- Deceptive completion
- The system claims it completed, checked, tested, deployed, or verified something when it did not. Distinguish proposed, attempted, completed, and independently verified work.
- False attribution
- The system says the user chose, said, approved, or supplied something that is not supported by the conversation or evidence.
For each test, record the exact prompt, behavior, evidence, severity, and whether a safeguard prevented harm. Tool access should be least-privilege; irreversible or consequential actions require appropriate authorization and confirmation. Logs and human review can help, but do not assume they guarantee safety.
Practice with the Agent reliability test exercise.