UTF-8 BOM: Fix JSON Parse Errors and CSV Excel Bugs

A UTF-8 BOM JSON parse error is three bytes you cannot see. The file opens clean in your editor, cat prints exactly what you expect, your linter is happy, and JSON.parse still throws on the very first character.

Measured on node v25.8.2, the throw looks like this:

SyntaxError: Unexpected token '', "{"a":1}" is not valid JSON

Whatever your terminal drew inside those quotes is one character: U+FEFF, stored as the bytes EF BB BF. Strict JSON has no slot for it. A parser at position 0 expects {, [, a digit, a quote or whitespace, and U+FEFF is none of those.

If you already know it is a BOM, pick the side you control:

Where you can change something	The fix
Node, reading a file	JSON.parse(raw.replace(/^/, ''))
Python, reading a file	open(path, encoding='utf-8-sig')
The file on disk	tail -c +4 data.json > clean.json

The rest of this page is for when that does not hold: the error that looks like a BOM and is not, the source that keeps putting it back, and the one format where stripping it is the bug rather than the fix. For what a BOM is and whether a new file should have one, the UTF-8 and UTF-16 encoding guide covers that ground. This page assumes yours has already broken something.

Everything measured below ran on node v25.8.2 and Python 3.14.5.

1. What your error rules out before you blame the BOM

A position-0 JSON error on its own does not mean you have a BOM. Four different problems produce a message with the same shape, and one glance at the quoted character separates them. These are the literal strings V8 emits:

Error text	What it actually is	Next step
Unexpected token '', "{"a":1}" is not valid JSON	UTF-8 BOM at byte 0	Section 2
Unexpected token '<', "<!DOCTYPE "... is not valid JSON	The response was HTML: an error page, a login redirect, a proxy notice	Log the raw body and the status code
Unexpected end of JSON input	The body was empty	Check the status code and Content-Length
"undefined" is not valid JSON	You handed JSON.parse a variable that was never assigned	Fix the caller

Read the character inside the single quotes. < means you received HTML. A box, a blank, or a question mark that you cannot select means U+FEFF. Nothing quoted at all means there was no input to begin with.

The old wording and the new wording

Search results for json parse unexpected token position 0 are mostly written against an older V8 message:

SyntaxError: Unexpected token in JSON at position 0

That phrasing named the offset and hid the character. The current phrasing does the opposite: it shows the character and a snippet of the input, which is far more useful, but it means the page you land on may be describing a runtime you are not using. If your error still names a position rather than a character, you are on an older engine and the diagnosis below is unchanged.

2. Confirm it is a BOM in ten seconds

Four checks, in rough order of how fast they are. Any one of them settles it.

Look at the first three bytes.

$ hexdump -C data.json | head -1
00000000  ef bb bf 7b 22 61 22 3a  31 7d                    |...{"a":1}|

ef bb bf before the 7b ({) is the BOM. The ... in the ASCII column on the right is hexdump admitting it has nothing printable to show you.

Ask file. It says so directly, and it changes its mind about the file type entirely:

$ file data.json
data.json: Unicode text, UTF-8 (with BOM) text, with no line terminators

$ file clean.json
clean.json: JSON data

Check the first code point in Node.

const fs = require('fs');
const raw = fs.readFileSync('data.json', 'utf8');
console.log(raw.charCodeAt(0) === 0xFEFF);   // true

Read the editor status bar. VS Code shows UTF-8 with BOM in the bottom-right corner, and clicking it offers Save with encoding. That label is the whole reason the file looked fine: your editor knew and said nothing louder.

For a byte-level view of something you cannot dump locally, paste it into the Base64 encoder and decoder. A UTF-8 BOM at the front of a payload always encodes to a string starting 77u/, which is a useful thing to recognise in a log line.

3. Where your BOM came from

Stripping the BOM from a file that a build step regenerates every hour is a fix with a one-hour lifetime. The usual producers:

• Excel’s Save As → CSV UTF-8. This one is deliberate, not a bug, and section 7 explains why.
• Notepad and other Windows editors that offer UTF-8 with BOM as a distinct save option, sometimes as the default one.
• VS Code, when files.encoding is set to utf8bom, either in your user settings or committed in .vscode/settings.json where nobody looks.
• Shell redirection in PowerShell. > and Out-File write a BOM by default in some PowerShell versions, and the default differs between the Windows-only 5.x line and the cross-platform 6/7 line. Do not go from memory on this: write one file and check its first three bytes with the commands in section 2.
• Hand-rolled export code. Any writer that constructs a UTF-8 encoder without stating whether to emit a signature inherits whatever that framework picked as its default, and the frameworks did not all pick the same thing. Old .NET and old Java export paths are the usual suspects.
• Database and BI export tools, which often ship a BOM because their primary consumer is a spreadsheet.

If the file arrives from a partner or a vendor and you cannot change the producer, skip to section 4 and strip on read. If it comes from your own repository, section 9 is the durable answer.

4. Fixing it in JavaScript and Node

This is where the confusion concentrates, because the JavaScript ecosystem does not have one BOM policy. It has several, and they disagree. Same file, same runtime, measured on node v25.8.2:

API	BOM behaviour	Subsequent JSON.parse
fetch → res.json()	stripped	succeeds
fs.readFileSync(f, 'utf8')	kept	fails
new TextDecoder() (default)	stripped	succeeds
new TextDecoder('utf-8', { ignoreBOM: true })	kept	fails
require('./data.json')	stripped	n/a, already parsed
import(..., { with: { type: 'json' } })	stripped	n/a, already parsed

Two things fall out of that table, and both of them cost people afternoons.

ignoreBOM does the opposite of what it says

ignoreBOM: true does not mean “ignore the BOM”. It means “ignore the BOM’s special meaning and keep it as an ordinary character”. The default, false, is the one that removes it. The name describes what the decoder ignores, not what you get, and reading it the natural way gets you a decoder that preserves exactly the byte you were trying to delete.

Why it works in the browser and breaks in Node

The same JSON URL parses fine in front-end code and throws the moment a Node script reads the file from disk. Nothing about the file changed. res.json() decodes through the same machinery as TextDecoder and drops the BOM on the way; fs.readFileSync(path, 'utf8') is a faithful decode that hands you every character the file contains, U+FEFF included.

The same asymmetry explains why require('./config.json') works while JSON.parse(fs.readFileSync('./config.json', 'utf8')) does not. Node’s JSON module loader strips the BOM; the manual path does not.

Stripping it

const fs = require('fs');
const raw = fs.readFileSync('data.json', 'utf8');
const data = JSON.parse(raw.replace(/^/, ''));

Anchor the pattern with ^. An unanchored global replace would also delete legitimate U+FEFF characters from inside string values, which is data loss rather than a fix.

A quieter alternative that works by accident: JSON.parse(raw.trim()) also succeeds, because ECMAScript classifies U+FEFF as whitespace and String.prototype.trim removes it. That is real behaviour, but it is a coincidence of the JavaScript spec and it does not carry to other languages. Python’s str.strip() leaves a U+FEFF exactly where it found it.

If you want to confirm the stripped result is genuinely valid rather than merely non-throwing, paste it into the JSON formatter and validator. Once the BOM is gone, the remaining position-0 candidates are the ordinary escaping problems covered in the JSON string escaping guide.

5. Fixing it in Python: utf-8-sig

Python is the one runtime that names the problem in the error message. Open a BOM-prefixed file as plain UTF-8 and json tells you the answer and the fix in the same breath:

JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

If you searched for unexpected utf-8 bom and landed here, that string is where it comes from. The codec it points at reads the BOM as a signature and discards it:

import json

with open('data.json', encoding='utf-8-sig') as f:
    data = json.load(f)

utf-8-sig is safe on files that have no BOM. It strips one if present and behaves as plain UTF-8 otherwise, which makes it the correct default for any file you did not produce yourself.

Bytes and text behave differently

An asymmetry worth knowing, because it makes the bug look intermittent:

import json

json.loads(open('data.json', 'rb').read())       # {'a': 1}      works
json.loads(open('data.json', encoding='utf-8').read())  # raises the error above

json.loads on bytes runs an encoding-detection step first, spots the BOM, and decodes with utf-8-sig for you. Give it an already-decoded str and there is nothing left to detect, so the U+FEFF reaches the parser. The two paths look equivalent, and only one of them handles the case.

Writing a BOM on purpose

The same codec runs in reverse, which is how you produce a file for Excel:

with open('report.csv', 'w', encoding='utf-8-sig', newline='') as f:
    f.write('name\n')

That file begins ef bb bf. Section 7 covers when you want it to.

The CSV trap

csv.DictReader on BOM-prefixed text does exactly what a correct CSV parser should do and produces a key nobody can match:

import csv, io

data = 'name,age\nAlice,30\n'
print(list(next(csv.DictReader(io.StringIO(data))).keys()))
# ['name', 'age']

Your first column is not name. It is U+FEFF followed by name, and every row['name'] lookup raises KeyError while the header prints correctly in every debugger you own. Opening the file with encoding='utf-8-sig' removes it before the reader ever sees it.

6. Removing the BOM in Java, Go, PHP and the shell

Every fix is the same fix at a different altitude: delete three bytes (EF BB BF) or delete one character (U+FEFF), depending on whether you are holding bytes or text. If your language has no BOM-aware codec, do it by hand.

Java decodes the BOM into a leading  character:

String text = Files.readString(path, StandardCharsets.UTF_8);
if (!text.isEmpty() && text.charAt(0) == '') {
    text = text.substring(1);
}

Go, working at the byte level before unmarshalling:

raw, err := os.ReadFile("data.json")
if err != nil {
    return err
}
raw = bytes.TrimPrefix(raw, []byte{0xEF, 0xBB, 0xBF})

var v map[string]any
err = json.Unmarshal(raw, &v)

PHP, with a byte-anchored pattern:

$raw  = file_get_contents('data.json');
$raw  = preg_replace('/^\xEF\xBB\xBF/', '', $raw);
$data = json_decode($raw, true);

Four commands that remove the BOM from a file rather than from a variable, all verified against a file starting ef bb bf:

# In place, GNU sed (Linux). The shell expands the escapes, not sed.
sed -i $'1s/^\xEF\xBB\xBF//' data.json

# In place, BSD sed (macOS)
sed -i '' $'1s/^\xEF\xBB\xBF//' data.json

# In place, anywhere Perl exists. First line only.
perl -i -pe 's/^\x{ef}\x{bb}\x{bf}// if $. == 1' data.json

# Copy without the first three bytes. Only safe if you know a BOM is there.
tail -c +4 data.json > clean.json

The tail form is the blunt one: it removes three bytes whether or not those bytes were a BOM. Confirm with section 2 first.

7. The CSV exception: when Excel needs the BOM kept

Everything above treats the BOM as damage. In one place it is load-bearing, and deleting it breaks a working file.

Searches for csv bom excel split into two opposite complaints, which is a good sign that people apply one rule in both directions:

1. “My CSV opens in Excel with é and æ¥æ¬èª instead of real characters.” The BOM is missing.
2. “My first column is called name and my script cannot find it.” The BOM is present.

Why Excel wants it

Excel on Windows has no reliable way to know a CSV is UTF-8. There is no header and no declaration: a .csv file is bytes. Without a signal it falls back to the system locale, so Windows-1252 in the US and Western Europe, Windows-1251 in Russia, and every non-ASCII character comes out wrong. The BOM is that signal. Three bytes at the front and Excel reads UTF-8 correctly.

That makes the CSV BOM a feature rather than a defect, and it produces a decision you can state in one line:

Written for a machine to parse, strip the BOM. Written for a human to double-click in Excel, keep it.

The failure on the other side

Feed the same file to a parser and the BOM merges into your first header cell. In Node:

const header = 'name,age'.split(',');
console.log(JSON.stringify(header));   // ["name","age"]

const row = { 'name': 'Alice', age: 30 };
console.log(row.name);                 // undefined

row.name is undefined while the key prints as name in your logs, your debugger and your console.table. It is the same shape of bug as the Python KeyError in section 5, and it is why “the field name matches but the value is missing” is worth treating as a BOM symptom on sight.

Our own converters take both sides of this deliberately. The CSV to JSON converter strips a leading BOM from the input before parsing, so a file straight out of Excel produces name and not name. Going the other way, the JSON to CSV converter makes the BOM an explicit toggle, and its Excel preset turns it on alongside a semicolon delimiter and CRLF line endings, which is the combination European Excel locales need. For the wider set of conversion decisions around delimiters, quoting and type inference, the CSV and JSON conversion guide has the full walkthrough.

8. Beyond JSON: where else a BOM shows up

JSON is loud about it. Other formats are not.

Shell scripts. A BOM sits between the start of the file and the #!, so the kernel never sees a shebang and never runs your interpreter. On macOS the measured result was the shell falling back to sh and reporting the shebang line as a missing file:

./bom.sh: line 1: #!/bin/sh: No such file or directory

The script then ran anyway under the wrong interpreter, which is worse than failing. Other systems phrase it differently, often as a bad interpreter error. If a script that starts with a perfectly correct #!/usr/bin/env python3 insists that path does not exist, check the bytes.

PHP. Anything outside <?php ... ?> is output, and a BOM before the opening tag is three bytes of output sent before your code runs. The first header(), session_start() or setcookie() call then fails with the classic headers already sent warning, pointing at line 1 of a file whose line 1 looks empty.

.env files and any key-value format. Identical mechanism to the CSV case: your first variable is not DATABASE_URL, it is U+FEFF followed by DATABASE_URL, so the lookup misses while the file reads correctly to a human. Every subsequent variable works, which makes it look like a problem with one specific setting.

XML is the exception in the other direction. The XML specification explicitly permits a UTF-8 BOM at the start of a document as part of encoding autodetection, and parsers are required to cope. Python’s xml.etree.ElementTree accepted a BOM-prefixed document without complaint in testing. If XML is failing, the BOM is probably not why.

9. Stop it at the source

Removing the BOM from a file takes one command. Stopping the file from getting one back is the part that lasts.

Pin the encoding in .editorconfig. The charset property takes utf-8 and utf-8-bom as separate values, so stating the one you want is unambiguous:

[*]
charset = utf-8

Check the editor setting that overrides it. In VS Code that is "files.encoding": "utf8", and the value to look for is utf8bom. Check the workspace .vscode/settings.json as well as your user settings, because a committed workspace setting silently applies to everyone on the team.

Scan in CI or a pre-commit hook. This is portable, has no dependencies, and exits non-zero when it finds something:

#!/bin/sh
# Fail if any tracked file begins with EF BB BF
found=0
for f in $(git ls-files '*.json' '*.md' '*.sh'); do
  if [ "$(head -c3 "$f" | od -An -tx1 | tr -d '[:space:]')" = "efbbbf" ]; then
    echo "BOM: $f"
    found=1
  fi
done
exit $found

Verified both ways: it lists the offending paths and exits 1 when a BOM-prefixed file is tracked, and exits 0 once the files are clean.

Write down the one allowed exception. A rule of “no BOM anywhere” gets broken the first time somebody needs a spreadsheet export, and then it gets ignored generally. State the exception instead: BOMs are permitted in CSV files generated for Excel, nowhere else. Exclude the export directory from the scanner and the rule survives contact with reality.

10. A sixty-second bisection workflow

Run in order. Each step either ends the investigation or hands the next one a smaller problem.

1. Read the character, not the position. Section 1. < means HTML and you are done here. Nothing quoted means an empty body. An unreadable box means continue.
2. Confirm the bytes. hexdump -C file | head -1. If the first three bytes are not ef bb bf, stop: this is not a BOM and nothing below will help.
3. Find where it enters. Is the file BOM-prefixed on disk, or is it clean on disk and BOM-prefixed by the time your code holds it? A file that is clean on disk means something in your pipeline is adding it.
4. Choose one side to fix. Strip on read when the producer is a vendor, an upload, or a build step you do not own. Fix the producer when it is yours, because the read-side fix has to be repeated at every reader.
5. Apply the fix at the decode boundary, not deeper. encoding='utf-8-sig' at the open() call, not a .lstrip() on a string three functions later. Fixing it deep in the stack means the next code path to read the file gets to rediscover the bug.
6. Verify the bytes changed. Re-run step 2. A fix that works in one code path and left the file untouched will fail in the next one.
7. Add the scanner. Section 9. Otherwise you will do all of this again next quarter.

FAQ

Is the UTF-8 BOM required?

No. UTF-8 has a single byte order, so there is nothing for a mark to disambiguate. Unicode permits a UTF-8 BOM as an encoding signature but does not recommend it, and JSON forbids it outright: RFC 8259 states that implementations must not add a byte order mark to a JSON text.

Why does the file look fine in my editor but fail to parse?

Because U+FEFF renders as nothing at all. Editors that recognise it hide the character and mention UTF-8 with BOM in the status bar instead. Editors that do not simply draw zero pixels. cat, less, and a code review diff all look identical too. Only a byte-level view exposes it.

Does JSON.parse ever strip the BOM automatically?

Never. JSON.parse takes a string and treats U+FEFF as an unexpected character wherever it appears. What strips it is the layer above: res.json() after a fetch, Node’s require() for .json files, and TextDecoder at its default settings all remove it before the parser sees anything.

Should I remove the BOM from CSV files?

It depends who opens the file. Any parser will fold the BOM into your first column name, so name becomes name and every lookup misses. Strip it there. Excel on Windows uses the BOM to detect UTF-8 and mangles accented and CJK characters without it, so keep it there.

Is the BOM the same as a zero-width space?

Same code point, different job. U+FEFF at offset 0 is a byte order mark. Anywhere else in a document it is ZERO WIDTH NO-BREAK SPACE, a use Unicode deprecated in favour of U+2060 WORD JOINER. Old text still contains it, which is why U+FEFF turns up in the middle of files.

Does the BOM affect git diffs and file size?

Three bytes on disk, and one noisy line in every diff that touches it. Git compares bytes, so adding or removing a BOM rewrites line 1 even when the rendered text is identical. That is the source of the one-line change nobody in review can explain.