Cleaning Windows-1251 and BOM Issues Out of a CSV Before You Import It

A file can look completely fine and still carry the wrong encoding

A CSV file exported from older accounting or database software is quite often not UTF-8 at all. Windows-1251 for Cyrillic text and Windows-1252 for Western European accented characters are both still common outputs from systems built years ago. A spreadsheet program will often display such a file without complaint, because it applies its own encoding guess automatically. So the file looks normal right up until it is read by something less forgiving, such as a script, an import pipeline or a database loader, that assumes UTF-8 and gets a wall of garbled characters instead of the text that was there.

A three-byte marker at the start of the file can break a strict parser

A UTF-8 byte order mark is an invisible marker, just three bytes, sitting at the very beginning of a file to signal that the text that follows is Unicode. It is helpful for a program like Excel that checks for it. It becomes a problem the moment the destination is a strict JSON importer or a script that doesn't expect it. Those extra bytes end up glued onto the first character of the first column header, so a column meant to be read as "id" no longer matches that name anywhere the rest of the code expects it. Detecting and removing that marker before conversion, rather than leaving it in the resulting file, avoids a mismatch that can be confusing to track down after the fact.

Getting the right encoding automatically, without a menu of options

Reliable encoding detection, the kind CSV to JSON runs before it reads a single row, checks for a byte order mark first, then tries to validate the remaining bytes strictly as UTF-8, and only falls back to Windows-1251 or Windows-1252, based on how the higher-value bytes are distributed, if that strict validation fails. That order matters. A file that is already UTF-8 is never second-guessed into being misread as something else, while a file that isn't UTF-8 gets identified correctly rather than being forced through as UTF-8 and turned into garbled characters along the way. The same detection sits in front of CSV to Excel, for the cases where the destination is a spreadsheet rather than a script.

Data types matter as much as the character encoding does

Getting the character encoding right solves only half of what breaks during a CSV-to-JSON conversion. The other half is deciding which values should become JSON numbers or booleans and which should stay as plain text. A value like 00123 turning into the number 123 loses its leading zero for good the instant that conversion happens, and an identifier longer than about fifteen digits gets rounded off the moment it becomes a standard floating-point number instead of staying as text. Both should be left as strings, because converting them changes the data itself, not just its formatting or appearance.

A stray duplicate header can break an import differently than a bad encoding does

JSON objects cannot hold two properties with the same key. So a source file with two columns that happen to share a header name, which is not unusual in exports assembled from more than one source system, needs those duplicates renamed to something distinct during conversion. Otherwise the second column's data overwrites the first one's in every resulting object, without any warning. That is a separate failure mode from encoding or typing, but it is just as capable of quietly corrupting an import. It is worth checking that a conversion tool reports when it happens rather than resolving the collision invisibly, and running the result through JSON Formatter shows the renamed keys at a glance.

Step by step: checking the file before it goes into the import

Convert the export once in CSV to JSON with the delimiter and the encoding left on automatic, then check both guesses: send the result on to JSON Formatter, one of the next steps the page offers, and read the first few records. Garbled Cyrillic or broken accents mean the encoding guess was wrong; a single long key with the separators still inside it means the delimiter guess was. Set the wrong one by hand and convert again: the encoding menu offers Windows-1251, Windows-1252, UTF-8 with or without BOM and UTF-16 LE, the delimiter menu comma, semicolon, tab, pipe or a character of your own. A file whose separator or encoding cannot be made sense of at all is refused with a message rather than converted into nonsense.

Next, decide how values are typed: the default already leaves values with a leading zero and long identifiers as text, and for a file made of codes, phone numbers or account numbers, setting value types to keep everything as text converts nothing at all. While the result is open in JSON Formatter, look for two traces of a messy source: two columns both headed id come out as id and id_2, and a row with more fields than the header carries the extras under an __extra key. Both are cheaper to fix in the source file than in the system that would choke on them later. Up to twenty files, 300 MB in total, go through in one batch without leaving the browser tab.