Real Life Use Cases for Structured Data Querying using LLM

Ever felt like data analysis is speaking a whole different language? You're not alone. Those tangled SQL queries can turn even the bravest analyst into a gibberish-spewing mess.

But fear not, fellow data wranglers! There's a new sheriff in town, and its name is LLaMA (no, not the fuzzy kind). LLama, the Large Language Model, is here to translate your plain English questions into perfect SQL queries. That's right, you can finally ditch the cryptic code and ask questions like, "Show me all the customers who bought stuff last week and forgot their wallets." (Hey, it happens!)

Here's how LLaMA can be your data dream come true:

• Business Intelligence Bonanza: Imagine asking your database, "What are our top-selling products?" in plain English and getting a clear answer. LLama makes it possible for anyone, not just SQL superstars, to unlock valuable insights.

• Customer Support Superhero: Customer service reps can ditch the database deciphering and use LLaMA to quickly find answers to customer questions. "Show me all open tickets for Karen in the hat department" - easy peasy!

• Healthcare Hero: Researchers can ask natural language questions about patient data, like "What's the average age of diabetic patients?" This can save time and potentially lead to better healthcare outcomes. Just imagine, doctors with less time wrestling with databases and more time healing!

• E-commerce Einstein: Businesses can ask LLaMA, "Who are our repeat customers in California?" and get the info they need to target the right people with the right offers. More sales, more happy dances!

Pinterest Engineering–Success Story

The Text-to-SQL Challenge: At first, we tested our system on a benchmark dataset. It worked well, but these tasks were simpler than real-world problems.

Real People, Real Data: Once users started using our Text-to-SQL tool, we saw a big improvement! The system got better at understanding user questions, and users were able to write SQL queries much faster with AI help (around 35% faster).

Data Quality Matters: We also found that clear and organized data made the system work even better. The more information there was about the data tables, the easier it was for the system to find the right info.

Boston Consulting Group

They were initially excited about the Yale Spider Challenge, a popular benchmark for text-to-SQL. Teams were achieving high accuracy rates, suggesting the problem was solved! However, when we tested our system with real-world data, things went south. The Spider dataset is great for academic research, but it doesn't reflect the messy reality of business data.

Real companies have inconsistent data structures, confusing column names, and all sorts of other issues that the Spider dataset doesn't account for. This mismatch between the academic world and the real world led to a big drop in performance for our system.

Open-source models were like training wheels for text-to-SQL. They worked okay, but they weren't as powerful as commercial models with tons of training data (think 1.5 trillion parameters!). To make our system smarter, we created a special technique that injects business knowledge and database details into the model's prompts. This, combined with additional training, helped it handle complex requests better.

Measuring success wasn't easy. Standard accuracy scores didn't capture the full picture. We needed a system that considered both if the answer was technically correct and if it made sense to a human. So, we developed a combination of metrics, including checks for correct execution, human-like evaluation using another large language model, and comparisons to see how close the answer was to what a person might expect.

Conclusion

Text-to-SQL is a game-changer for businesses by making data analysis accessible to everyone. Forget complex SQL code! Now anyone can ask questions of the company's data using plain English. This empowers marketing teams to understand customer trends, sales reps to personalize outreach, and even customer support to answer questions faster.

No more waiting for specialists to write cryptic queries –– text-to-SQL puts the power of data analysis in everyone's hands. This translates to faster decision-making, improved customer service, and ultimately, a competitive edge. Text-to-SQL isn't just about convenience, it's about unlocking the true potential of your data.

Check out our other piece for a deep dive into Text to SQL here

Sources

Pinterest Engineering