Artificial Intelligence
What It Takes to Turn User Ideas into AI-Generated Designs
Introduction
Over the past six months at OmbuLabs.ai, we’ve had the opportunity to work on several projects involving generative AI for designing and enhancing real world products.
With today’s image generation models, it might seem like this should be straightforward. A user describes what they want, an image generation model creates a design, and you’re done.
In practice, building an application that consistently produces useful product designs is much more involved. The image generation model is only one part of the solution. The real challenge is designing the application around the model so that it has the information and context it needs to generate useful results.
There is no single way to build these kinds of systems. Every project presents different challenges, different sources of domain knowledge, and different technical constraints. Throughout development, we found ourselves making a series of decisions. Where should different types of information live? Should it become part of a prompt? Should it be retrieved from a knowledge base? Should it be provided through a tool or reference image? Which image generation model best supports the workflow we want to build?
In this post, we’ll walk through how we approached those decisions. We’ll look at the system that transforms a user’s request into a design that better reflects a company’s products, expertise, and constraints.
The Illusion of “Easy” AI Design
To make this concrete, let’s imagine a company called SneakerCompany. They are a direct-to-consumer brand that wants to let customers design their own sneakers using AI.
A customer enters a request like:
“A modern sneaker with a premium feel, neutral tones, and subtle texture.”
An image model generates a sneaker, and you’re done, right?
If the goal is simply to create an interesting image, maybe.
If the goal is to generate a product that could actually be manufactured, that single request needs to become something much richer.
That richness includes a material for the upper (leather, knit, suede), a call on whether the sole is minimal or performance focused, and a concrete definition of what “premium” means for this company. It also has to reflect which materials are approved, which combinations can actually be manufactured, and which finishes align with the brand.
This is the gap between AI-generated images and real product design. Success depends less on the image model itself and more on the system that communicates with the model to generate the right result.
Capture Domain Knowledge
Once we understood the problem we were trying to solve, the next step was understanding what information an application would need.
That knowledge rarely comes from a single source. Depending on the product, it may come from designers, engineers, manufacturing teams, product managers, brand specialists, existing documentation, product catalogs, or years of institutional knowledge.
Each source contributes a different piece of the puzzle.
Some of the questions we started with included things like:
- How do experts describe the product when speaking about it?
- What terminology do they use?
- Which combinations should never be generated?
- What materials, colors, or finishes are approved?
- What information exists in a visual form?
- What do users commonly ask for, and how was the company interpreting that before?
As we worked through these questions, we realized that not all knowledge is created equal.
Some information rarely changes, like brand guidelines or materials that should never be used.
Some information is highly visual, such as stitching patterns, sole designs, or shoelace styles.
Other information changes regularly, such as product catalogs, trends or historical customer designs.
Understanding the different kinds of knowledge available became just as important as collecting the knowledge itself.
Designing the System
Once we understood the information available to us, the next challenge was deciding how the application should use it.
There is no single architecture for AI-powered design applications. Every piece of information raises a different design decision.
While iterating with our clients we began coming up with these kinds of questions for ourselves:
- Should a business rule, like a banned sole material, become part of the instructions sent with every request?
- Should thousands of historical sneaker designs be retrieved from a database when needed?
- Should curated images of approved laces and stitching patterns be provided directly to the image generation model as reference images?
- Should the application call a tool to retrieve information that changes frequently, like current shoe trends?
These decisions weren’t made in isolation. They were influenced by both the type of information we were working with and the capabilities of the image generation model. As we refined the system, we found ourselves iterating between the architecture and the model. Sometimes a new piece of domain knowledge changed how we wanted to build the application. Other times, the capabilities of the model changed what was possible.
That kind of back-and-forth iteration is only possible if you can see what’s actually happening inside the system, which is why we built tracing in as early as possible. AI applications often have many moving parts, including retrieved context, reference images, tool calls, prompts, and model responses. Having visibility into each step made it much easier to understand why a particular design succeeded or failed, and let us iterate on the system with much greater confidence.
For example, we found that company-wide rules, such as sneaker materials that should never appear in generated products, were a natural fit for the system prompt because they should apply to every request.
On the other hand, large collections of historical product designs were too extensive to include directly in a prompt. Instead, they were better suited for retrieval or tool calls that could surface the most relevant examples based on the user’s request.
Highly visual concepts presented another challenge. Curated reference images of approved laces, materials, or stitching patterns often produced more reliable results than text descriptions alone. In those cases, the workflow was designed to provide both text and images to the model.
As we gathered more domain knowledge, we were continually asking the same question:
What’s the best way for the application to use this information?
That question shaped nearly every architecture design decision we made.
Selecting an Image Generation Model
As we designed the system, we quickly realized that many of our architectural decisions depended on the capabilities of the image generation model itself. Choosing a model wasn’t simply a matter of finding the one that produced the nicest images. We needed to understand what each model could do, how those capabilities fit into our workflow, and whether the tradeoffs made sense for the application we were building.
One example came from a project where we wanted to generate four different design concepts for the user on the very first iteration. This immediately raised the question of whether the model could generate multiple images in a single request, and whether doing so together would be more efficient than issuing four separate requests, given the tradeoffs each approach carried for latency and cost.
Cost became an important consideration because image generation pricing varies significantly between models. A workflow that generates four images for every user interaction can become expensive very quickly, making it important to balance image quality with the overall cost of running the application. We keep the client updated on these costs and how they could change over time as we are going through the project.
Input capabilities also played a major role in our thought process. Some models only accept text, while others can work with one or more reference images. As we discussed in the previous section, we found that certain types of domain knowledge, such as approved materials, stitching patterns, or historical product designs, were often communicated more effectively through images than text alone. That meant we needed a model capable of accepting image inputs as part of the generation process.
Editing capabilities were another important factor. In many design applications, the first image is only the beginning of the conversation. Users often want to refine an existing design by changing the color, replacing a material, or adjusting part of the product while leaving everything else unchanged. If the workflow depends on iterating from previous designs, the model needs to support image editing in addition to generating images from scratch.
Rather than selecting a model based on benchmarks alone, we found it more useful to evaluate models against realistic product design workflows. Image quality was only one consideration. We also compared cost, latency, support for reference images, editing capabilities, and how well each model handled the kinds of inputs our application would generate. The architecture, the available domain knowledge, and the desired user experience all influenced which model was ultimately the best fit.
Building and Refining the System
Designing the architecture and selecting an image generation model wasn’t the end of the process. Building AI-powered applications is inherently iterative, and many of our early assumptions changed as we gained a better understanding of the domain and observed how the system behaved in practice.
As we experimented with different approaches, we continually refined how information was presented to the models. Small changes to the instructions sent with each request, retrieved context, reference images, or tool outputs could have a significant impact on the quality of the generated designs. Rather than thinking about prompt engineering as a single activity, we found it more useful to think about how the entire system prepared information for the model.
Tracing played an important role throughout this process. Because image generation applications involve many moving parts, including prompts, retrieved knowledge, tool calls, reference images, and model responses, it can be difficult to understand why a particular result was successful or where something went wrong. Building tracing into the application from the beginning gave us visibility into every step of the workflow and made it much easier to evaluate architectural changes as the system evolved.
Evaluation became equally important. Instead of only evaluating the final image, we looked at the entire workflow: whether the correct reference images were retrieved, whether business rules were applied correctly, and whether the generated design reflected the user’s original request. Looking at the complete system helped us identify opportunities for improvement that would have been difficult to find by evaluating the model alone.
Putting It All Together
Let’s return to our SneakerCompany example.
A customer enters the request:
“A modern sneaker with a premium feel, neutral tones, and subtle texture.”
Rather than sending that request directly to the image generation model, the application begins gathering the additional information needed to produce a useful result.
It applies company-wide business rules, retrieves similar historical sneaker designs, selects curated reference images for approved materials and laces, and incorporates any other relevant context gathered during the design process. The application then combines all of this information into the final request that is sent to the image generation model.
The model uses that enriched context to generate design concepts that better reflect the company’s products, design language, and manufacturing constraints rather than relying solely on the user’s original request and a model’s breadth of knowledge.
Although every application will differ, the overall workflow often follows a similar pattern.
The exact implementation will vary depending on the product and the domain, but the underlying principle remains the same. The application is responsible for transforming a simple user request into the information the image generation model needs to produce useful, realistic, and brand-appropriate designs.
Conclusion
One of the most interesting parts of working with generative AI for physical products is realizing that success rarely comes from the model alone. The real value comes from understanding the people already doing the work, capturing the knowledge they rely on every day, and building systems that translate that expertise into something scalable and usable.
As image generation models continue to evolve, the biggest opportunities won’t simply come from generating better images. They’ll come from designing better systems around those models, systems that combine domain expertise, business rules, reference material, and user requests into a workflow capable of producing reliable and useful results.
Have an idea for how AI can enhance your product? Our team at OmbuLabs.ai can help. Let’s chat .