The Challenge

Understanding user interactions at scale

As their AI products grew in popularity, Gamma found it difficult to understand user experience and debugging issues at scale. Some pain points that Gamma’s product team experienced included:

• Lack of visibility. Lacking a deep intuition and understanding of how their users engage with their LLM-powered products.
• Difficulty tracking errors. The large amounts of unstructured data generated by LLMs made it difficult to track down bugs and which errors were most common.
• Segmented data sources. Existing tools couldn’t tie together metrics, errors, and traces in one place, making it difficult to get the full context around a user interaction (for debugging).

We felt like we were flying blind regarding how some of our core features performed in production. There are error lookup tools like Sentry that let me investigate one error message in-depth, and analytics tools like Amplitude that show overall usage. However, I haven't seen a tool to tie those together for the Gen AI workflow.

The Solution

Collaborative GenAI product workspace

Through a simple SDK, Autoblocks enabled Gamma to seamlessly integrate LLM product optimization into their workflow. It quickly became the de facto GenAI product workspace at Gamma, enabling their leadership, product managers, designers, and engineers to collaborate on making GenAI products better.

With Autoblocks, Gamma is able to efficiently wrangle and interpret troves of unstructured GenAI product data (using Autoblocks' search, filtering, and evaluation capabilities). When anomalies or errors were identified, Autoblocks was instrumental in understanding every step of the LLM pipeline, including API calls, prompt chaining, and pre- and post-processing.

Autoblocks not only accelerates debugging but also empowers the Gamma team to make proactive product improvements. By analyzing real-world usage data in Autoblocks, they better understand user needs. Then, armed with these insights, they collaboratively prototype test changes within the Autoblocks platform.

User insights inform iterative product improvements, propelling this virtuous cycle of GenAI product refinement. Soon after implementing Autoblocks, Jon says his team saw a major improvement in product quality.

It gave me the full picture of what happened during each user interaction, which helps us keep improving our GenAI products in a thoughtful way.