Happy to see new versions popping up! I hope VUO still has a bright future ahead: )
Would it even be possible to integrate ChatGPT to perhaps write a prompt of what to create? Or at least get a head start on a project…
When you say “write a prompt”, are you thinking it would generate a suggestion on what to make (a phrase or sentence of text, similar to a creative writing prompt)? Or were you thinking something more involved?
I was thinking more in terms of the LLM to have insight and knowledge of witch nodes to incorporate, and therefor also set them up so that you would get a great starting point for your project, or even better, a final solution. Kind of how https://makereal.tldraw.com creates mockups of apps, from images and text → working prototype
: )
Not sure if you’ve given the current ChatGPT 3.5 a try but it already has a fair bit of knowledge regarding Vuo and its nodes (up to Jan 2022). It’s able to recommend possible node suggestions to help solve a proposed problem. I only played with it a couple times, asking some fairly basic questions to test it out. I found its answers were fairly accurate and if you go into it understanding that it might hallucinate a bit it could still be a useful in working through a problem.
It was more accurate than what it suggested for a simple Blender script to make a linked chain. That one took 10 tries and me finally suggesting a specific way to do it before it was finally able to get close to what I asked. Haha! It did however do a nice job of reading a badly written and slightly broken script; it cleaned it up, simplified it, and fixed the broken part. That I was actually quite impressed by, though I did have to tell it what the script was supposed to be doing.
Have also used GPT to help plot coordinates in vuo. Would be cool to see some sort of LLM nodes. Local LLM or API.
@cwilms-loyalist and @krezrock, interesting to hear of your experiences asking ChatGPT for help with Vuo. If either of you happen to have the time & energy to make a longer writeup or video, I’m guessing people would be quite interested to see some working (and not-working) examples.
Coming back to the feature request — if it gets a significant number of votes, we can check the state of the technology at that point and see how difficult it would be. With ChatGPT 3.5, we were able to obtain a set of steps that came close to solving a small and specific problem (“Create a Vuo composition that increases the contrast of live video”). But there’s a gap between the ChatGPT output and a working composition, which includes translating the steps from English text to an actual Vuo composition and working around mistakes such as the wrong node/port names that ChatGPT emitted during our test. Those are things a human can do well (we’re quite fault-tolerant) but would be hard to automate in terms of writing an integration between Vuo and ChatGPT (if ChatGPT’s instructions aren’t precisely correct in all details, it may not produce a meaningful or even syntactically correct composition). ChatGPT or its successors would have to fill more of the gap for this to be feasible. There’s also the question of how well ChatGPT would do with more complex and interesting problems.
As another avenue toward providing starting points for projects, I would also mention that the example compositions are meant to help with this — so if you have any ideas for examples that you think would be commonly useful, please feel free to create a feature request or share a composition you’d like to see incorporated into Vuo.
Hi Jaymie,
My use of Chat GPT is very limited. My use involved asking it to help solve some integration issues with Ableton and Touch Designer. And plotting some grid points for me in Vuo coordinates. It’s really hit or miss, and it’s questionable if it’s helpful or not.
Sorry for the late reply, I’ve honestly only played around with ChatGPT for a few minutes regarding Vuo. I would say in my very limited testing it had about 90-95% correct responses. Some of the things it got wrong were suggestions to “double click” nodes to configure the settings. I keep my ChatGPT history pretty pruned, so I don’t have the original conversation any more.
A few of the things I remember asking it were:
- What is Vuo? It got this correct and had a pretty decent explanation if I recall.
- I asked it to explain, in layman’s terms, the function of a few random nodes just to see what it actually knew. I remember its responses being quite accurate. But to be fair my questions weren’t that deep
- I went a little deeper and asked it some more complex questions. “Can you explain to me the relationship between Vuo composition units and pixel co-ordinates?” It gave a very thorough and detailed answer breaking down how Vuo used a Cartesian coordinate system with 0,0 being the centre of the composition space and that Pixel were based on the display resolution and DPI with 0,0 being in the top right corner etc. It then proceeded to explain what considerations were needed in order to place elements where you want on screen. One thing I remember it didn’t get right was its example using a position in Vuo coordinate of 50,50 to place an object in the upper right corner of the screen, yep it was in the upper right corner alright!
- I also asked it a couple questions on what nodes might be useful for solving specific types of issues and to provide examples of how to connect and use them. The suggestions of which nodes to use were pretty on-point, but the examples of how to connect and use them were sometimes a little off showing it understood what the node did but not always how the node did it. Also any details about input values were almost never accurate so it clearly had no concept at this point of what a nodes values actually meant and is was just guessing.
I believe it could serve as a valuable tool for overcoming a mental block, suggesting potential nodes that someone might not have considered. However, when it comes to solving a specific problem, it is not a quick and straightforward solution.
1 Like
In theory, one could finetune a GPT3/4 model with a dataset generated by Vuo examples. You could create pairs composed by the example’s description and the composition code. The finetuned model would then create a new composition given a prompt that ask for a specific, new problem to solve. Technically is something not too difficult to achieve, I don’t know how effective this could be; the only chance would be to try. I wonder if Vuo license allows using examples code to create such a model.
Vuo’s example compositions are shared under the MIT License, which would allow using them to train a model as long as the compositions generated from that model contain the same copyright notice and permission notice as the originals.