Benchmarking Predictabl on RelArena

Over the past days we've been benchmarking Predictabl on RelArena, Prior Labs' benchmark for prediction on relational data: 21 tasks across 7 real-world databases, alongside relational transformers, graph neural networks and TabPFN-based methods.

Using RelArena's official training data and evaluation protocol, Predictabl ranked:

• 🥇 1st on 4 of 21 tasks
• 🏆 Top 5 on 11 of 21 tasks

Predictabl handled the ML pipeline automatically. An LLM framed each prediction and wrote the features as readable, auditable SQL. Predictabl then prepared the data, checked for leakage, and trained and validated LightGBM models on CPUs.

No manual feature engineering. No task-specific tuning.

A few things we learned

• Bigger models don't always win. Automatically generated features and a simple LightGBM model running on CPUs can compete with sophisticated deep learning architectures, without expensive GPU infrastructure.
• The model is only one part of the job. Understanding the database, identifying useful signals, generating features, preventing data leakage, training and validating models: Predictabl automated the entire workflow, not just the final prediction step.
• One approach can work across very different problems. From Amazon product lifetime value and H&M customer churn to F1 podium finishes and repeat event attendance, the same automated pipeline finished 1st without engineering a solution by hand for each task.

Building our own training rows

We also took the experiment a step further. Instead of using RelArena's prepared training rows, we let Predictabl build its own directly from the raw databases, choosing which entities and dates to learn from. A step the other benchmark methods didn't have to handle.

Here, it reached the top 5 on 8 of 21 tasks, and its own training rows performed better than the official ones on 5 tasks.

Full study coming soon. 🧪