Following up on my full review of Luma Dream Machine (see my previous post for the deep dive), I wanted to share a quick tutorial on a specific workflow that tripped me up initially: transforming subjects in an existing video.
I had a great clip of two animal curators, Britt and Gregory, in a forest and thought, “Why not turn them into the animals they care for?”
The Goal: Video-to-Video Transformation
For those who missed the first post, Luma is a generative AI tool. While it’s famous for creating videos from scratch, its “Modify” feature allows you to upload existing footage and use AI to hallucinate completely new textures and subjects over the original motion.
I wanted to keep the curators’ movements and the forest setting, but replace their human forms with realistic lemurs.
(Insert the final Lemur video result here)
The Breakthrough Workflow
My initial attempts failed because the AI didn’t know how to map the animals onto the people correctly. Here is the exact recipe that finally worked:
1. Create a “Start Frame” with Gemini This was the key unlock. Luma needed a visual hint. I took a still screenshot of Gregory and Britt from the video and a separate photo of lemurs. I used Gemini to merge them into a single composite image—basically a “mock-up” of what I wanted the first frame to look like.
I uploaded this composite image into the “Start Frame” slot in Luma. This gave the AI a perfect reference for where to start.
2. Use the “Modify” Tab with a Strong Prompt I uploaded the original video into the Modify tab and used a very specific text prompt to ensure the camera didn’t go crazy:
“Replace the people with two realistic lemurs standing in the forest, furry textures, lemur faces, wildlife photography style, static camera, fixed shot, no zoom, no camera movement.”
3. Modify the Strength Slider “Adhere to structure and performance details” was the lowest strength setting, but it didn’t work—it actually showed Gregory’s neck. I changed it to the middle setting, “Flex the structure and performance creativity,” and that got the final result. There is also a “Full Creative” setting (Reimagine) and a couple of options in between those.
A Warning on Credits!
Be careful with your resolution settings. I didn’t realize until it was too late that generating a 7-second clip in 1080p costs over 1,000 credits.
If you are just testing a new prompt or workflow, stick to 540p—it only costs a couple 50 credits. Save the high definition for when you know the prompt works!
Finally, I have to give a shout-out to Gregory and Britt. Thank you for being excellent test subjects! No humans or lemurs were harmed in the making of this video.
As an FYI, I leveraged Gemini very substantially as an interactive manual on how to do this, how to answer questions and finally, how to put this into a final blog – all the info was there. I used my trial and error to easily share more info on the process.
