Categorical vs Numerical Features: Choosing Representations That Models Can Use

The real question is not whether a column contains text or numbers. It is whether the model should treat those values as ordered quantities or as separate labels.

Why the Same Column Can Fool a Model

Imagine a dataset with a color column that stores red, green, and blue as the integers 1, 2, and 3. The column looks numerical. It contains digits. You could even compute its average and get 2.0, which feels like a meaningful summary.

It is not. What is the average of red and blue? The question does not make sense, because color is not a quantity. It is a label. The integers 1, 2, and 3 are just a compact way of writing three category names.

This is the trap that catches most beginners: numbers in a cell do not make a numerical feature. The distinction between categorical and numerical features is about meaning, not storage format.

Here is the mental test I use: does arithmetic make sense on this column? If averaging two values produces something you could never encounter in real life, you are probably looking at categories wearing a numeric costume.

Before we go further, one quick bridge: in supervised learning, features are the columns the model reads to make predictions, and the target is what you ask it to predict. This article is about how each feature type should be represented before the model ever sees it.

What Makes a Feature Categorical or Numerical

Categorical features hold labels or groups. The values are categories, not measurements. Numerical features hold quantities where math genuinely works: you can average them, compare differences, and compute meaningful results.

But each type splits further, and the split matters for encoding.

Nominal categorical features have no meaningful order. Color, city, product type, and blood type are nominal. Sorting them alphabetically does not reveal anything about the data.

Ordinal categorical features have a meaningful order, but the gaps between categories are uneven. T-shirt size (small, medium, large) and education level (high school, bachelor's, master's, PhD) are ordinal. You know medium sits between small and large, but you cannot say the gap from small to medium equals the gap from medium to large.

Discrete numerical features are countable quantities. Number of rooms in a house, number of children, number of website visits. These often take whole-number values.

Continuous numerical features are measurements that can take any value within a range. Height, temperature, price, and time all qualify.

Type	Subtype	Example	Does order matter?	Does arithmetic make sense?
Categorical	Nominal	Color, city, product type	No	No
Categorical	Ordinal	T-shirt size, education level	Yes	No
Numerical	Discrete	Number of rooms, visit count	Yes	Yes
Numerical	Continuous	Height, temperature, price	Yes	Yes

The tricky case is ordinal categorical. Values like small, medium, and large can be ranked, which makes them feel numerical. But the ranking does not turn them into measurements. The gap between small and medium is not guaranteed to equal the gap between medium and large.

What Happens When You Feed Raw Categories to a Model

Most scikit-learn estimators expect numbers. They cannot consume text labels like "red" or "Seattle" directly, so every categorical column must be converted into some numeric representation before training.

The naive approach is to map each category to an integer: red becomes 1, green becomes 2, blue becomes 3. This is often called label encoding or integer encoding. It works in the narrow sense that the model can now read the column. But it quietly teaches the model something false.

A linear model or a distance-based model sees the integer 2 as a magnitude. It assumes green is twice as far from red as blue is from red, and that the value 1.5 would sit halfway between red and green. None of that is true. The model has invented a quantity where only labels exist.

Tree-based models tolerate integer-coded categories better because they split on thresholds rather than computing distances. A tree can learn that values less than 2.5 behave one way and values greater than 2.5 behave another. But the arbitrary order still creates odd split boundaries, and the model can waste capacity learning patterns that exist only because of your numbering scheme.

The encoding choice matters because it changes what the model believes about the data. Feed a model a column of integers and it will treat them as quantities. The question is whether that belief matches reality.

One-Hot Encoding: Giving Each Category Its Own Column

The standard solution for nominal categories is one-hot encoding. Instead of one column with integers, you create one binary column per category. Each row gets a 1 in the column for its category and 0 everywhere else.

Red becomes [1, 0, 0]. Green becomes [0, 1, 0]. Blue becomes [0, 0, 1].

This removes the false order entirely. No category sits between any other. The distance from red to green is the same as the distance from red to blue, which is exactly what you want when the categories have no inherent ranking.

The tradeoff is width. A column with 10 categories becomes 10 columns. A column with 500 categories becomes 500 columns. The feature matrix grows wide, training cost rises, and on small datasets the extra columns can hurt more than they help.

This problem has a name: high cardinality. When a categorical column has hundreds or thousands of unique values, one-hot encoding explodes the feature matrix. Scikit-learn's OneHotEncoder handles the mechanics, and you can set handle_unknown='ignore' so unseen categories at prediction time do not crash your pipeline. But the dimensionality problem remains.

One-hot encoding is a strong fit for linear models and distance-based methods, where false order is actively harmful. It is often wasteful for tree-based models, which can usually work with simpler encodings.

Ordinal Encoding: When Order Is Real

Ordinal encoding maps ordered categories to integers that respect their rank. Small becomes 1, medium becomes 2, large becomes 3. This is legitimate when the order is real and meaningful.

The catch is that the model assumes equal spacing between ranks. An ordinal encoding of education level treats the gap between high school and bachelor's as identical to the gap between bachelor's and master's. That assumption is usually wrong, but it is far less wrong than pretending the categories have no order at all.

Tree-based models are more forgiving of this assumption because they split on thresholds rather than using the numeric spacing directly. Linear models take the spacing literally, so ordinal encoding is riskier for them.

My rule: use ordinal encoding when order is real and meaningful, and use one-hot encoding when categories are unordered. If you are unsure whether the order is real, ask whether a domain expert would rank the values without hesitation. If the answer is no, treat the feature as nominal.

Choosing an Encoding for the Model, Not Just the Data

Here is the insight that ties everything together: the best representation depends on the model family, not just on the data type. The same categorical column may deserve one-hot encoding for a linear model and ordinal encoding for a gradient-boosting model.

Linear models and distance-based methods interpret every number as a magnitude. They need one-hot encoding for nominal categories, and they benefit from scaled numerical features so that no single column dominates the distance calculation.

Tree-based models and gradient boosting split on thresholds. They can often work with ordinal-encoded categories, and they generally do not need numerical features to be scaled at all. A tree does not care whether a feature ranges from 0 to 1 or 0 to 1000; it only cares where the useful split points are.

Model family	Nominal categories	Ordinal categories	Numerical features
Linear models, distance-based methods	One-hot encode	One-hot encode or ordinal encode with caution	Scale
Trees and gradient boosting	Ordinal encode or one-hot encode	Ordinal encode	Leave as-is

This pattern shows up in real scikit-learn workflows. A neural network preprocessor typically one-hot encodes categorical features and scales numerical features. A gradient-boosting preprocessor often leaves numerical features untouched and ordinal-encodes categorical features. The data did not change. The model did, so the representation changed with it.

Common Mistakes and How to Recover

These mistakes are normal. I have watched beginners make all of them, and the recovery is always a small, repeatable check.

Mistake 1: Treating a numeric-coded category as a numerical feature. The color column with values 1, 2, 3 gets scaled or fed directly into a linear model. The symptom is a model that learns nonsense relationships. The fix is to recognize the column as categorical and encode it appropriately.

Mistake 2: One-hot encoding a high-cardinality column. A city column with 800 unique values becomes 800 columns, and your training time balloons. The symptom is a wide, sparse feature matrix that slows everything down. The fix is to reconsider whether the column belongs in the model at all, or whether you can group rare categories into an "other" bucket.

Mistake 3: Ordinal-encoding nominal categories. Mapping red, green, and blue to 1, 2, 3 teaches a linear model that green is the average of red and blue. The symptom is a model that performs well on training data but fails in ways you cannot explain. The fix is to ask whether the order is real before choosing the encoding.

Mistake 4: Forgetting unseen categories at prediction time. Your training data contains three cities, so your encoder knows three categories. Then a new row arrives with a fourth city, and your pipeline crashes. The symptom is an error at prediction time that you never saw during training. The fix is to configure your encoder to ignore unknown categories or to handle them explicitly.

The Decision Rule You Can Reuse

Every time you prepare a dataset, run the same three-step check.

First, name each column's type. Is it categorical or numerical? Ignore the storage format and ask what the values actually mean.

Second, if it is categorical, ask whether order is real. If yes, ordinal encoding is on the table. If no, one-hot encoding is the safer default.

Third, choose the encoding that matches your model family. Linear and distance-based models need one-hot encoding for nominal categories and scaled numerics. Trees and gradient boosting can often work with simpler encodings and no scaling at all.

The next step is to build a repeatable preprocessing pipeline that applies these choices consistently across your training and prediction data. That is where the real work lives: not in memorizing encoding rules, but in building a system that encodes every column correctly every time.