As a follow up to my testing of https://labs.heygen.com/video-translate (Steve In Italiano, DDMC) I went back to Eleven Labs which I had tested previously for it’s incredible voice cloning ability since Heygen doesn’t support Chinese (yet).
What I found most interesting about the Eleven Labs new “dubbing” feature is that you don’t have to upload a video. You can simply provide it a link.
I tried giving it a Warpwire link and it unfortunately didn’t work – even when setting the permissions public and making it available for download. I wanted to wait until I had the translation quality tested before digging into the Warpwire connectivity issue. I ended up downloading the video
Here is the original video:
And in under 5 minutes, it delivered this video dubbed in Chinese:
Here are comments from William Wan from Duke Kunshan University:
“It is amazing in terms of translation accuracy in Chinese, it also generates tone as if a westerner speaks Chinese. One drawback is talking speed varies because it tries to match sentences (the number of words are different in two languages). Lack of lip sync is another one. But nevertheless, the software is amazing.”
To take it one step further, Todd Stabley loaded the Chinese translation into Panopto and we were able to generate Chinese closed caption:
https://duke.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=dd82e89c-6ac0-4177-a799-b0990138b51e
If this is something that people feel that they’d like to use in production, I’ll initiate a review by Duke’s security office.
