Artificial Intelligence Applications Technology Data Science Editors Pick Knowledge Graphs Machine Learning Staff Tutorials Tabular data—structured information stored in rows and columns—is at the heart of most real-world machine learning problems, from healthcare records to financial transactions. Over the years, models based on decision trees, such as Random Forest , XGBoost , and CatBoost , have become the default choice for these tasks. Their strength lies in handling mixed data types, capturing complex feature interactions, and delivering strong performance without heavy preprocessing. While deep learning has transformed areas like computer vision and natural language processing, it has historically struggled to consistently outperform these tree-based approaches on tabular datasets. That long-standing trend is now being questioned. A newer approach, TabPFN , introduces a different way of tackling tabular problems—one that avoids traditional dataset-specific training altogether. Instead of learning from scratch each time, it relies on a pretrained model to make predictions directly, effectively shifting much of the learning process to inference time. In this article, we take a closer look at this idea and put it to the test by comparing TabPFN with established tree-based models like Random Forest and CatBoost on a sample dataset, evaluating their performance in terms of accuracy, training time, and inference speed. What is TabPFN? TabPFN is a tabular foundation model designed to handle structured data in a completely different way from traditional machine learning. Instead of training a new model for every dataset, TabPFN is pretrained on millions of synthetic tabular tasks generated from causal processes. This allows it to learn a general strategy for solving supervised learning problems. When you give it your dataset, it doesn’t go through iterative training like tree-based models—instead, it performs predictions directly by leveraging what it has already learned. In essence, it applies a form of in-context learning to tabular data, similar to how large language models work for text. The latest version, TabPFN-2.5, significantly expands this idea by supporting larger and more complex datasets, while also improving performance. It has been shown to outperform tuned tree-based models like XGBoost and CatBoost on standard benchmarks and even match strong ensemble systems like AutoGluon. At the same time, it reduces the need for hyperparameter tuning and manual effort. To make it practical for real-world deployment, TabPFN also introduces a distillation approach , where its predictions can be converted into smaller models like neural networks or tree ensembles—retaining most of the accuracy while enabling much faster inference. Comparing TabPFN with Tree based models Setting up the dependencies Copy Code Copied Use a different Browser pip install tabpfn-client scikit-learn catboost Copy Code Copied Use a different Browser import time import numpy as np from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score # Models from sklearn.ensemble import RandomForestClassifier from catboost import CatBoostClassifier from tabpfn_client import TabPFNClassifier To run the model, you require the TabPFN API Key. You can get the same from https://ux.priorlabs.ai/home Copy Code Copied Use a different Browser import os from getpass import getpass os.environ['TABPFN_TOKEN'] = getpass('Enter TABPFN Token: ') Creating the dataset For our experiment, we generate a synthetic binary classification dataset using make_classification from scikit-learn. The dataset contains 5,000 samples and 20 features, out of which 10 are informative (actually contribute to predicting the target) and 5 are redundant (derived from the informative ones). This setup helps simulate a realistic tabular scenario where not all features are equally useful, and some introduce noise or correlation. We then split the data into training (80%) and testing (20%) sets to evaluate model performance on unseen data. Using a synthetic dataset allows us to have full control over the data characteristics while ensuring a fair and reproducible comparison between TabPFN and traditional tree-based models. Copy Code Copied Use a different Browser X, y = make_classification( n_samples=5000, n_features=20, n_informative=10, n_redundant=5, random_state=42 ) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) Testing Random Forest We start with a Random Forest classifier as a baseline, using 200 trees. Random Forest is a robust ensemble method that builds multiple decision trees and aggregates their predictions, making it a strong and reliable choice for tabular data without requiring heavy tuning. After training on the dataset, the model achieves an accuracy of 95.5% , which is a solid performance given the synthetic nature of the data. However, this comes with a training time of 9.56 seconds, reflecting the cost of building hundreds of trees. On the positive side, inference is relatively fast at 0.0627 seconds, since predictions only involve passing data through the already constructed trees. This result serves as a strong baseline to compare against more advanced methods like CatBoost and TabPFN. Copy Code Copied Use a different Browser rf = RandomForestClassifier(n_estimators=200) start = time.time() rf.fit(X_train, y_train) rf_train_time = time.time() - start start = time.time() rf_preds = rf.predict(X_test) rf_infer_time = time.time() - start rf_acc = accuracy_score(y_test, rf_preds) print(f"RandomForest → Acc: {rf_acc:.4f}, Train: {rf_train_time:.2f}s, Infer: {rf_infer_time:.4f}s") Testing CatBoost Next, we train a CatBoost classifier, a gradient boosting model specifically designed for tabular data. It builds trees sequentially, where each new tree corrects the errors of the previous ones. Compared to Random Forest, CatBoost is typically more accurate because of this boosting approach and its ability to model complex patterns more effectively. On our dataset, CatBoost achieves an accuracy of 96.7% , outperforming Random Forest and demonstrating its strength as a state-of-the-art tree-based method. It also trains slightly faster, taking 8.15 seconds, despite using 500 boosting iterations. One of its biggest advantages is inference speed—predictions are extremely fast at just 0.0119 seconds, making it well-suited for production scenarios where low latency is critical. This makes CatBoost a strong benchmark before comparing against newer approaches like TabPFN. Copy Code Copied Use a different Browser cat = CatBoostClassifier( iterations=500, depth=6, learning_rate=0.1, verbose=0 ) start = time.time() cat.fit(X_train, y_train) cat_train_time = time.time() - start start = time.time() cat_preds = cat.predict(X_test) cat_infer_time = time.time() - start cat_acc = accuracy_score(y_test, cat_preds) print(f"CatBoost → Acc: {cat_acc:.4f}, Train: {cat_train_time:.2f}s, Infer: {cat_infer_time:.4f}s") Testing TabPFN Finally, we evaluate TabPFN, which takes a fundamentally different approach compared to traditional models. Instead of learning from scratch on the dataset, it leverages a pretrained model and simply conditions on the training data during inference. The .fit() step mainly involves loading the pretrained weights, which is why it is extremely fast. On our dataset, TabPFN achieves the highest accuracy of 98.8% , outperforming both Random Forest and CatBoost. The fit time is just 0.47 seconds, significantly faster than the tree-based models since no actual training is performed. However, this shift comes with a trade-off—inference takes 2.21 seconds, which is much slower than CatBoost and Random Forest. This is because TabPFN processes both the training and test data together during prediction, effectively performing the “learning” step at inference time. Overall, TabPFN demonstrates a