generate_lipsync

pro.aicut/aicut · Verify this server

Takes a picture of a person and an audio file, and generates a video of THAT person speaking or singing the audio, lip-synced. Returns the new job's id immediately. The result is exactly as long as the audio. WHEN: the user wants a talking avatar, a presenter, a voiceover put on a face, or a portrait to say something. The picture supplies the person; the audio supplies both the performance AND the length. The audio must be a `generate_audio` result on this account - pass the id that tool returned, or its url. An external audio url cannot be used here yet, so if the user has audio elsewhere, the way in is to make it with `generate_audio` first. THE CARD: this call has ALREADY put the aicut card in front of the user. It shows the prompt and the model straight away - plus the aspect ratio and a correctly-shaped empty frame for a video or an image, while AUDIO has neither and simply shows the prompt until the player appears - and it fills in the finished media on its own, so you do not have to do anything to make that happen. Do NOT call `show_generation` for this job: the card is already in the conversation and a second call would put a duplicate one beside it. `show_generation` is only for bringing back an EARLIER generation the user asks to see again. HOW TO WAIT: call `wait_for_generation` with this job's id and its `media`. It waits SERVER-SIDE for up to 15 seconds and opens no card. If it answers `terminal: false`, wait `poll_after_seconds` and call `wait_for_generation` again with the same arguments - keep doing that until it answers `terminal: true`, then tell the user in ONE short line what came out. Do not poll `get_video` / `get_image` / `get_audio` in a loop instead, and do not spawn timers or background shells to wait. Say nothing between waits: do NOT tell the user the generation is running and that you will fetch the result when they ask - just wait, then tell them it is done. IF YOU CANNOT RENDER AN AICUT CARD - a terminal, a plain SDK client, anything that did not negotiate the MCP Apps UI extension - then no card appeared here, so give the user the `url` from the wait result directly. aicut cannot see which clients render cards and sends the same answer to all of them. TIMING: a video usually finishes in 1-3 minutes, and can take up to about 6. Do not assume it failed before then, and do not spawn timers or background shells to wait: call `wait_for_generation` (it waits server-side for up to 15 seconds per call), wait its `poll_after_seconds` and call it again, and stop when it answers `terminal: true`. `get_video` is a single immediate read for one job's raw fields - it is not the waiting loop and must not be polled in one. COST: this spends the user's tokens, and the price is the AUDIO's length times a per-second rate that depends on the resolution - so a two-minute voiceover costs many times a ten-second one, and 1080p costs twice 720p. The audio's length is MEASURED BY THE SERVER from the file itself, never taken from anything you say - that measurement, rounded UP to the next whole second, is the price and the charge. Audio the server cannot measure is REFUSED with nothing charged. There is deliberately no length argument on this tool. Pass `estimate_only: true` first to get the exact price. An EXTERNAL url is downloaded and stored on the account before it can be measured, which uses one of the account's daily media-ingest slots. Quoting and then creating from the SAME url costs that once, not twice - aicut reuses what it already imported - and a url already on the account costs nothing at all. SPEND ETIQUETTE (the money grammar): in the webapp the priced button is the user's own finger; in chat YOUR tool call is not - so state the price IN THE SAME MESSAGE as the ask, and the user's explicit go is the button press. Never charge on inference: quoting is not asking, and after a price you wait for the yes. THIS APPLIES TO EVERY TOOL CARRYING THIS NOTE, including this one. A GO IS SCOPED TO ONE PURCHASE, AND IT MUST BE UNAMBIGUOUS. The user's instruction has to NAME the thing you are about to buy, or refer to it so plainly that it cannot mean anything else. A BARE AFFIRMATION - 'go', 'yes', 'ok', 'do it', 'just do it', 'sure' - counts ONLY when ALL THREE of these hold: the message immediately before it was YOUR priced ask for THAT EXACT action, nothing else was raised in between, and NOTHING THE USER ASKED FOR EARLIER IS STILL OUTSTANDING. That last one is the trap the others miss: if the user's OWN previous turn asked for something else - a refusal, a different scene, an edit, a redraw, a question - their 'just do it' may be answering THAT, and it is AMBIGUOUS even when your priced ask happens to be the last thing said in the thread. An ambiguous affirmation is not a go: ask WHICH one they mean and state that price again. WHEN IN DOUBT ABOUT WHAT A 'GO' REFERS TO, ASK. A wrong guess spends the user's money on something they never asked for, and nothing on this surface can undo it or give it back - asking costs one sentence. An episode's STAGES - cast portraits, episode create, fire, render - are each their own priced ask. A STANDING GO IS NOT UNLIMITED: 'just make it' or 'go ahead with the whole episode' authorizes the stages you PRICED IN THAT SAME MESSAGE, in the order you named them, and nothing beyond them - so do not re-ask per stage while it holds, and do not stretch it over a stage whose price the user never saw. IT EXPIRES THE MOMENT THE USER RAISES ANYTHING ELSE - a change, a question, a refusal, a redraw, a new idea - and after that the next stage needs its own priced ask. ONE STAGE IS NEVER COVERED BY A STANDING GO AT ALL: the FIRE (`fire_story_video`) is irreversible and the biggest single charge in the episode, so it always takes a go that NAMES firing, whatever was said earlier - see that tool's own note. A REDRAW IS NOT A STAGE: `regenerate_story_frame` and `regenerate_cast_portrait` are extra spends the user asks for one at a time, so state that price every time, even under a standing go. A standing go never carries to a different episode, and never to `generate_video`, `generate_image` or `generate_audio` - each of those is its own ask. ACCOUNT FOR YOUR OWN CALLS: if the user says something happened that you did not intend - a charge they did not expect, a step they did not ask for - RE-READ YOUR OWN TOOL CALLS IN THIS CONVERSATION before you answer, and tell them plainly which tools you called and when. NEVER SPECULATE ABOUT A CAUSE YOU CANNOT OBSERVE: not a button on an aicut card, not the user's own click, not their client. The aicut cards CANNOT SPEND - the only tools they ever call are the reads (`get_video` / `get_image` / `get_audio`), and their buttons either save a file or send a VISIBLE user turn into the chat - none of them calls a spending tool - so saying a card might have generated or charged something is false, not a hedge. (If a spend followed one of those visible turns, it was still YOUR call, and the honest answer names it.) If your call history disagrees with what you told the user, say what you actually called and let them correct you; do not invent an explanation that makes the two agree. IDEMPOTENCY: `idempotency_key` is optional and makes a retry safe. Set it on the FIRST call, not only on a retry - the job is addressed by the key, so a key added afterwards cannot find a job that was created without one. Reusing a key REPLAYS the job that key already created and returns it unchanged - even if you send a different prompt or different settings, and even after that job has finished. A key is therefore spent permanently. Do NOT reuse one to make another generation: two deliberate generations are two jobs and need two different keys (or none). ONE EXCEPTION, on `render_story_video`: replaying a key whose render FAILED answers 409 `render_failed` rather than replaying the failure, because a spent key stays spent - retry that one with a NEW key or with none. A KEY IS NOT SCOPED TO A TOOL: it addresses a job on the whole account, so reusing the key you gave `generate_video` on `generate_story_video` replays that first video instead of starting an episode. One key, one thing you made. (`render_story_video` is the one door that namespaces its own, which is why an episode's key can be reused on its render without colliding - but there is no reason to reuse it there either.) Never derive the key from the request body. You do NOT need to pass one to be safe against a duplicated delivery: aicut already derives a per-call key server-side, so a retry the transport makes on its own replays rather than charging twice. Pass your own only when YOU want to retry a call whose answer you never saw. OUTPUT: this returns JSON for you to read. When you report back to the user, give them the media URL plus a one-line summary. Do not paste the raw JSON, job ids, or internal field names into the conversation.