Experimenting with AI coding tools

Our frontend team has been using Cursor with its deep integration with leading models. For production development they've found it a real power-up in certain areas - a partner that takes the context of the codebase and can automate repetitive but complex tasks.

But we've also been fascinated by the opportunities for rapid prototyping and 'vibe coding' - where you lean almost entirely on the AI tool to write code based on prompting and natural language conversation.

My role as Innovation Director at Numiko is to keep us ahead of the curve with these new technologies, so as a test, I wanted to see how far I could get without actually writing code myself.

Way back in the mists of time when Flash was not yet a proscribed technology, I loved the ability to create on-screen almost at the speed of imagination. With these assistive technologies, I can really start to feel that flow again. What I found is that it can really increase the speed of the creative/experimental loop; a proof of concept using a library or system you aren't familiar with or don't know at all can take shape unbelievably quickly.

The key benefit is that it will build out the foundations and write working code using frameworks and libraries without having to grind through the setup details or documentation. Essentially, you can run before you walk - and then go back and learn to walk in a much more effective way, by seeing code that's directly applicable to your use case.

Here's a couple of examples, using Cursor and claude-3.5-sonnet (and latterly 3.7) - generally held to be one of the strongest models for coding.

First, I wanted to try creating an isometric 3D landscape. After a bit of set up as a Vue.js project, I started asking the Cursor Composer to create an island using the 3D JS library Three.js. I thought it needed some palm trees, some sharks circling, and a little robot, all of which Cursor added happily and quickly. With a few requests, the sharks actually pointed in the direction they were moving and would chase and attack the robot when it waded into the sea. Getting the robot to walk around and then changing the control scheme was almost instant; getting it to walk at the right elevation for its position on the island was slightly trickier; as a human, I could eyeball it in a way that Cursor couldn't, and that took a bit more coaxing. You can see it here! In the demo video, the user controls the robot's movements with their keyboard.

The second example was to create a toy implementation of something using generated depth maps of flat photos to create 3D point clouds and models. I'd started by using a Huggingface demo of the dedicated model called DepthAnything to generate the maps, which I had offline alongside the original photos. This time I started with a blank file, rather than Vue.js, and asked Cursor to create a Three.js project that would take an image and a depth map and render a point cloud. It achieved this pretty much from the first prompt.

From there, it added camera controls to pan and zoom; I experimented with trying different rendering for the points including its interpretation of gaussian splatting, before changing it to generate a 3D mesh model instead. Adding more images and a picker made me wonder whether it was possible to generate the depth maps in browser; I googled a bit and found an implementation on github, which I gave to Cursor and asked it, casual-like, to rewrite the whole thing using this approach instead.

This was a slightly bigger job, which needed me as the human to download the model file, put it in the right place, and then do some human-powered visual debugging of why the depth map was too small or the wrong aspect ratio, too squiggly, not squiggly enough, and so on. But within about 20 minutes, it was working again!

Adding a loading spinner was a single sentence. You can see the result here, with Howard (Numiko's Chief Canine Officer) and Dan as our models.

Caveats? It occasionally gets stuck, and if you use unclear language in your prompting, it can understandably misinterpret your intent and go off on a bit of a tangent. But it has largely been incredibly perceptive, even when I've been deliberately terse or colloquial. 'Now add sharks' worked surprisingly well. 'Make the ships orbit the equatorial regions in a swirly pattern', was a mix of specific and ambiguous which it pretty much nailed first time, in an earlier test.

While it is great at nearly ‘unattended’ creation, I’ve found the greatest benefit understandably comes when you understand what the code is doing, and you have a mental model of what's going on. When you're in the detail and nuance of shaping something, the more it feels like an extension of your thought process, where you are still directing or conducting, but the AI is filling in the gaps.

Overall, I felt like the flow of shaping the experience was fairly seamless without any real stumbling blocks. As I knew my way around local development practices, working on these projects locally was pretty straightforward, and then pushing to Git Hub and deploying from there to free Vercel hosting was also easy but it did require some coding and 'plumbing' knowledge. We've also been trying out services like Bolt.new and v0 from Vercel, which do pretty much all of that scaffolding for you, by nature of being fully hosted. If you don't have any git or local development experience, those might be the best route in.

I'd recommend trying this out even if you don't have a defined problem to solve - you can just play and visualise. It's compellingly fun and addictive!