Hey everyone,

I'm in the middle of a migration project and could really use some advice. I need to move a client's master product data, which includes tens of thousands of SKUs. I know this can typically be handled with a standard XLSX import, but the problem is the data from their old software is a complete mess.

Let me give you an example of what I'm dealing with.

Let's say they have Product A. It comes in 3 color variants and has two different barcodes (one for a single unit, and one for a package/box of 12).

The problem is how they've been logging this. Each variant and package type is treated as a completely separate product, sometimes with a different SKU or sometimes they put different barcode (custom barcode, not barcode from vendor). There are even duplicates from old mistakes. The database looks something like this:

• Product A Blue - Box of 12
• Product A Red - Box of 12
• Product A Green - Box of 12
• Product A - Single Unit (any color just gets thrown under this SKU)
• Product A Blue - Box of 12 (Old Barcode) (a data entry mistake from 3 years ago that was never fixed)

On top of this, their naming conventions are all over the place. For an item like a pen, it might be listed as "Ballpoint," "Pen," "B-Point," or just a typo of the brand name.

This isn't an isolated case. Just to get an idea of the scale, we ran a simple query to find items with the same name but different barcodes, and it returned over 9,000 results.

I'm honestly feeling a bit lost on how to even begin cleaning this up. Has anyone dealt with a data nightmare of this scale before? I'm trying to figure out the best approach to standardize and consolidate everything before importing it into the new system.

Any advice on strategy, tools, or just a general workflow would be massively appreciated. Thanks