We Benchmarked Two Background Removal AI Models. Neither One Wins.

If you want better results with background removal ai model benchmark, this guide explains the practical steps, common mistakes, and useful browser-based tools that make the process easier.

We wanted to upgrade the AI model behind our Background Remover. The obvious way to choose is to read the model cards, compare the headline accuracy figures, and ship whichever number is biggest.

We did the measurement instead, against two public datasets with human-annotated ground truth, and the result was that no single model was the right answer.

One model beat the other decisively on portraits and then failed catastrophically on ordinary objects.

Quick Takeaways

• Focus first on the setup.
• Apply the steps from this guide to improve background removal ai model benchmark without overcomplicating the workflow.
• Use Background Remover to turn this advice into action directly in your browser.
• Read How to Remove an Image Background for Free (Online, No Signup) if you want a related guide that expands on the same topic.

This article is the full record: the method, the numbers, a bug that silently invalidated our first round of results, and the routing approach we ended up shipping.

Every figure here was measured on our own hardware through the actual browser pipeline, not copied from a paper.

The Setup

Two candidate models, both permissively licensed so they can ship in a commercial product:

• ISNet, as distributed with @imgly/background-removal. This is what our tool already used. Two variants exist: fp16 at 84.1 MiB and fp32 at 168.0 MiB.
• ORMBG (schirrmacher/ormbg), Apache-2.0, 168 MiB fp32. An ISNet-family architecture retrained by its author.

Both run in the browser on onnxruntime-web using the WebAssembly execution provider.

Scoring is Intersection over Union (IoU) at an alpha threshold of 128, plus mean absolute alpha error, computed against the final downloadable cut-out rather than the raw model output.

That distinction matters: what a user receives has been through threshold and edge-refinement steps, and measuring the raw tensor would flatter both models.

The Bug That Invalidated Our First Results

If you would rather do this step in the browser than by hand, Image Editor handles it without a signup.

Our first benchmark round produced a clear verdict: ORMBG mis-segments real photographs and is not worth shipping. That verdict was wrong, and the reason is worth more than the benchmark itself.

Every image fed to a model must be normalized first, and our pipeline applied one formula to every candidate on the reasoning that they are all ISNet-family, so the mean must be 0.5.

Nobody read the models' own inference code. When we did, every single model differed, and each differed in a different way:

• ORMBG divides by 255 and subtracts no mean at all. Its inference.py is simply torch.divide(im, 255).
• ISNet as shipped by @imgly computes (x - 128) / 256 on the raw uint8 values, with no separate division by 255.
• MODNet divides by 255, then applies mean 0.5 and standard deviation 0.5, giving a range of -1 to 1, and uses a reference size of 512 rather than 1024.
• BEN2 and BiRefNet-lite both divide by 255 and then apply the ImageNet mean and standard deviation.

Scored against the ORMBG author's own published cut-out for his own example photograph, the difference is not subtle:

• With the wrong normalization: IoU 76.3 percent, mean absolute alpha error 35.6 out of 255, false negatives 10.0 percent.
• With the correct normalization: IoU 99.9 percent, mean absolute alpha error 0.8 out of 255, false negatives 0.0 percent.

The Benchmark: Portraits Versus Objects

With normalization fixed, we scored both models against two public datasets with human-annotated alpha channels. P3M-500-NP is a portrait matting set.

DIS-VD is a set of complex non-human objects. Results are IoU percentages for the final cut-out, higher is better.

P3M-500-NP, portraits

• ISNet: 92.4, 95.1, 98.2, 98.6, 96.0, 97.1. Mean 96.2.
• ORMBG: 98.9, 98.5, 99.0, 99.3, 99.4, 99.0. Mean 99.0. ORMBG wins all six images.

DIS-VD, objects

• ISNet: 99.3, 82.8, 95.1, 62.2, 96.2. Mean 84.9.
• ORMBG: 32.3, 16.7, 19.6, 8.3, 14.5. Mean 19.2. ORMBG collapses.

A mean IoU of 19.2 is not noise and it is not a tuning problem. On one image ORMBG marked 40.9 percent of the object as background; on another, 29.9 percent.

The explanation is straightforward once you look at what the model is for: ORMBG is trained on humans, and it segments humans. Hand it a chair and it does not know what it is looking at.

Why We Route Instead of Choosing

Image Upscaler is the quickest way to apply what this section describes to your own file.

The obvious response to these numbers is to send portraits to ORMBG and everything else to ISNet. That requires knowing whether the photo contains a person as its subject, which turns out to be a harder question than it sounds.

Our first attempt used pixel heuristics: colour distribution, corner uniformity, palette size. Across a twelve-image test set it classified 9 of 12 correctly and found only 3 of the 6 humans.

That is worse than useless for routing, because a missed portrait means the better model never runs.

The fix was a small object detector, NanoDet-Plus-m-1.5x at 3.6 MiB, running in 72 to 92 milliseconds once warm. With it, classification went to 12 of 12, all 6 humans found, and zero objects mistakenly sent to the human model.

One more measured surprise: we tested an int8 export of the detector, which is 3.7 times smaller than fp32. It ran at 391 milliseconds against fp32's 180.

The quantize and dequantize nodes deoptimize on the WebAssembly backend, so the smaller model is more than twice as slow. We ship fp32.

Where COCO Runs Out

Detectors only recognise the classes they were trained on, and the standard COCO label set has significant blind spots for e-commerce.

We ran five real Amazon listing photographs through the detector and three of them returned zero detections, because COCO has no class for a craft kit, a board game or a puzzle, and no class for clothing at all.

That is a property of the label set rather than a defect in the detector, and it is why the pixel heuristics are still in the pipeline as a fallback rather than being deleted. The two methods cover different failure cases.

On those same five photographs the heuristics correctly identified three as product shots from corner uniformity alone.

Two Performance Findings Worth Copying

The wrong ONNX Runtime bundle costs 10.7 MiB

onnxruntime-web ships several entry points and the file name does not tell you which runtime binary it pulls. Measured from the network log:

• ort.bundle.min.mjs loads ort-wasm-simd-threaded.jsep.wasm, at 22.8 MiB.
• ort.wasm.min.mjs loads ort-wasm-simd-threaded.wasm, at 12.1 MiB.

Both execute the same graph identically on the WebAssembly provider. We were using the bundled build while our own comments said 'WASM only', so every visitor downloaded a GPU runtime we deliberately do not use.

Check the network log, not the filename.

Cross-origin isolation is worth 39 percent

There is more detail in How to Edit Photos Online for Free Like a Pro (No Install) once you have the basics here.

WebAssembly threads require SharedArrayBuffer, which is gated behind cross-origin isolation. Without the COOP and COEP headers on the document, our runtime asked for 12 threads and silently received 1.

It printed a console notice rather than an error, so nothing surfaced it.

After enabling isolation, the same 1200 by 1200 image went from 15,659 milliseconds to 9,518 milliseconds, a 39 percent improvement, for a byte-identical cut-out.

If you run any WebAssembly workload in a browser, check window.crossOriginIsolated before you optimise anything else.

Honest Limitations

Three caveats we would want to see stated if we were reading someone else's benchmark:

• The 99.9 percent normalization figure was scored against ORMBG's own author's published output. That proves our pipeline reproduces the reference implementation; it does not prove ORMBG is accurate in general.
• The ground-truth masks from the datasets server are served as JPEG, which is lossy. Absolute alpha error is therefore a floor for both models. The comparison remains valid because both pay the same penalty.
• Six images per dataset is a small sample. It is enough to establish a 65-point gap and not enough to rank two models that are close together.

We also tested and rejected several models on runtime grounds rather than accuracy: BiRefNet-lite fp16 is refused outright by the WebAssembly provider, and its fp32 export loads and then throws during inference.

ORMBG cannot run on the WebGPU provider at all, which fails with a missing MaxPool kernel, so our tool defaults to WebAssembly and takes WebGPU only when explicitly asked.

What This Means If You Are Choosing a Model

• Benchmark on your own content, not on the dataset the model was trained on. The 65-point gap here is invisible in either model's published figures.
• Read the model's own inference code for its normalization. Do not infer it from the architecture family.
• Check the licence first. Two of the highest-scoring models available today cannot be used commercially at all.
• Measure the final output your user receives, not the raw tensor. Threshold and refinement steps change the number.
• A smaller quantized model is not automatically faster. Measure it on the runtime you actually ship.

Our Background Remover now detects whether the subject is a person and routes accordingly, which is why it handles both a portrait and a product photograph without you choosing a mode.

Everything runs on your own device: the models are downloaded to your browser and the image is never uploaded.