Data Quality¶
The Data Quality page (menu: Prices & Products → Data Status, address
/pim/data) shows how complete and correct the product data is across
every live product: overall, by supplier, by team and category, field by
field and rule by rule. Every figure opens the list of products behind it, so
the page is also a work list for fixing gaps. It replaced the earlier
completeness page from version 3.8.3.
Group users see every supplier and can build the figures. A supplier user who opens the page sees only their own company's products.
Which products are counted¶
Live products only: status Active or Special and not discontinued. Products on other sites, discontinued products and products with any other status are left out.
How a product is scored¶
Each product is checked against six sections of its record. A section's completeness is the share of its checks the product passes, and the quality score (0 to 100) is the average of the sections that are scored.
The sections, levels and rules below are eBiz's defaults. A site can change them; see Changing what is scored.
| Section | What is checked |
|---|---|
| Identity | Product name, brand or manufacturer, manufacturer part number, GTIN, range, supplier category |
| Pricing | A live price; invoice price, list price, RRP, an end date, VAT code, unit of measure and pack size on a live price |
| Logistics | Weight and its unit, length, width and height, dimension unit, pallet quantity, pallet GTIN, lead time, country of origin |
| Marketing | Web name, short description, long description, three or more feature bullets, search keywords, colour, material or finish |
| Media | A product image, two or more images, a product datasheet, a technical, safety or compliance document |
| Compliance | Certification, ETIM class, environmental information, packaging material weights |
A value counts as given unless it is empty or a placeholder such as N/A,
-, TBC or 0. A live price is one whose effective-from date has
passed and whose effective-to date has not, or that has no end date, in any
price zone.
Data Levels¶
Each product is also given a Data Level, by default from 1 to 5. The default levels follow Data Yard's five Data Levels as closely as the fields held here allow. A product is at the highest level whose requirements, and those of every level below it, it meets. A product that does not meet Level 1 is shown as Below Minimum (level 0).
| Level | Name | Requires |
|---|---|---|
| 1 | Minimum | Product name, brand, manufacturer part number, a product image, and no invalid GTIN |
| 2 | Core | Weight, length, width and height, pallet quantity, lead time, country of origin |
| 3 | Enriched | Long description, three or more feature bullets, web name, a datasheet or two or more images |
| 4 | Environmental | Environmental information, packaging material weights (see Packaging components) |
| 5 | Standardised | A valid ETIM class (the form EC000000) |
A GTIN is not needed for any level, because many heavyside products have no barcode. A GTIN that is given must be valid, though: a product with a GTIN of the wrong length or check digit stays below Level 1. This differs from Data Yard's own Minimum level, which asks for a GTIN. A missing GTIN still lowers the Identity section and shows as the warning No GTIN.
Validation rules¶
Every product is checked against these rules. Errors are data that is wrong; warnings are data that is missing or doubtful.
| Severity | Rule |
|---|---|
| Error | No product name |
| Error | Supplier part number also used by another of the supplier's products |
| Error | GTIN has the wrong length or check digit |
| Error | No live price |
| Error | Packaged weight is less than the product weight |
| Error | Pallet GTIN has the wrong length or check digit |
| Warning | Product name over 150 characters |
| Warning | No GTIN |
| Warning | GTIN also used by another of the supplier's products |
| Warning | GTIN also published by another supplier |
| Warning | Price ends within 30 days and no new price is loaded |
| Warning | Weight given without a unit |
| Warning | Only some of length, width and height given |
| Warning | Dimensions given without a unit |
| Warning | No product image |
| Warning | ETIM class is not in the form EC000000 |
| Warning | Not updated in over 12 months |
A GTIN must have 8, 12, 13 or 14 digits and a correct check digit. Spaces and
dashes are ignored. A GTIN that a spreadsheet has turned into a number such as
5.01234E+12 fails the check.
Product keys¶
A product is known by its supplier, supplier part number, manufacturer, manufacturer part number, alternative part number and GTIN. A supplier may publish products made by another manufacturer.
The supplier part number is unique within a supplier: it is the key a
product file is matched on. Case and outer spaces are ignored, so ABC-1 and
abc-1 are the same part number. Two suppliers may use the same part number.
On a product's Template tab the part number cannot be changed to one
another of the supplier's products has, or left empty. Products that already
share a part number are flagged so they can be put right.
A GTIN is flagged when another of the supplier's live products has it, and separately when another supplier publishes it. On the product's Template tab both show as warnings on the GTIN, naming the other supplier, and do not change the Data Level. The Template tab finds only copies stored the same way; the Data Quality build also finds copies stored with spaces or dashes.
The tabs¶
- Overview — the quality score and its change over 28 days, live products, the share at Level 3 or above, products below Level 1, products with errors and the share updated in the last 12 months; the spread of Data Levels; completeness by section; the score over the last 180 days; the rules most products fail.
- Suppliers — a scorecard per supplier: products, score and its change over 28 days, Data Levels, each section, products with errors and the share up to date, in a table that can be searched by supplier name, sorted by any column and paged. Click a supplier to filter the whole page to it. Filter Options in the panel heading narrows the list of suppliers: by Team and Category (suppliers with products there), by Score band, by Data Level (fewer than half the supplier's products reach it), by a Weak section (its average is under 50%), and with switches for suppliers that have products with errors, whose score fell in the last 28 days, or with under half their products up to date. A supplier must match every filter in force; the filters show as chips under the heading.
- Categories — the same figures by team and category. Teams and categories come from the agreement each product is loaded against: the team is the top tier of the agreement categories (the tier directly under the root group), and the category is the tier under the team. An agreement placed directly on a team has no category.
- Fields — the share of products with each field filled, with the number missing. Click a field for the products missing it.
- Issues — every rule with the number and share of products failing it.
- Products — the products behind whatever was clicked, worst score first, with their level and the rules they fail, in a paged table that can be searched by product code or name and sorted by product, supplier, score, level or last update. Export downloads them in the same search and order, with the fields each is missing, as a spreadsheet of up to 50,000 rows. Filter Options in the panel heading opens a box for each kind of filter, on top of whatever was clicked: Supplier, Team and Category (the same filters as the boxes at the top of the page, so the other tabs follow), Issues and Missing fields (each can be added more than once), Data Level, Incomplete section, and switches for With errors and Not updated in 12 months. A product must match every filter in force, so "No live price" plus "No product image" lists the products with both. The filters in force show as chips under the heading, each removable on its own, and the button shows how many there are.
The supplier, team and category filters at the top apply to every tab.
Saved reports, in the heading of the Products tab beside Export, keeps the page as it is under a name: the supplier, team and category, the tab, and the Products tab's filters. It is the same control the report builders use: a saved report is yours alone unless an administrator publishes it, when it shows as a button at the top of the Products list for everyone. The control appears where both the site and its PIM are on version 3.8.3. See Saved reports for how saving, sharing, publishing and defaults work.
When the figures are worked out¶
The figures are worked out every night where the nightly task is set up, and when a group user presses Build (or Rebuild once there are figures). Opening the page never starts a build. The time of the last build is shown at the top of the page.
The first build scores every live product. After that, a rebuild only scores again the products whose figures can have changed since the last build: products that were edited or had prices or media imported, prices that started, ended or came within 30 days of ending, products that passed 12 months without an update, new products, and products sharing a GTIN. All the others keep their figures. So a rebuild usually takes a minute or two even on a large catalogue. Full rebuild (in the menu next to Rebuild) scores every product again; it is only needed if something looks wrong, and a full build also runs on its own when the last one is more than a week old or the scoring rules have changed.
While a build runs the page shows what it is doing and how far it has got, and the figures from the previous build stay on screen until it finishes. A build carries on if you leave the page.
If a build stops part way, for example because the service restarted, it is marked as failed after ten minutes without progress, the reason is shown at the top of the page, and a new build can be started.
One Data Level, worked out here¶
A product has one Data Level, and it is the one this page's latest build worked out. The product's Template tab, the home dashboard and the supplier scorecards all show that figure.
Where a site has a product template of its own in which attributes say which Data Level needs them, the template defines the levels: the build asks the template for each product's level instead of using the levels in Site Settings. A product is then at the highest level for which every attribute that level and the levels below need is filled in and breaks no rule, which for the top level includes full ETIM classification where the template asks for it. The level names stay as set here, and each level's summary on this page lists the template attributes it needs. Changing the template, or the site's product data rules, makes the next build score every product again.
A site with no template of its own uses the levels described above.
Because the figure is stored by a build, a change to a product shows in its Data Level after the next build. The Template tab shows the stored level with the time of the build and, where the site's template defines the levels, what the level will be at the next build if the product has changed since.
The percentage bars on the product search and under the product's tabs are the older field completeness measure, not the Data Level. They stay for now and are labelled as such.
Changing what is scored¶
A site administrator can change what the page scores on, in Site Settings → PIM → Data Quality, and press Save Settings:
- Data Levels: rename a level, change the checks it needs, add or remove levels (one to five). Where the site's own product template defines the levels, only the names set here are used. A level's summary is made from its checks unless one is typed in.
- Sections: rename a section, or move checks between sections. Each check can be in one section only. A section with no checks is not scored and is not shown on the page. As well as eBiz's checks, a section can count any single product field (filled in) or any single price field (filled in on a live price, in any price zone). More price checks are offered than the defaults use: rebate price, a next price loaded, quantity breaks, discounts, unit of order and its quantities, pricing unit, pallet size and minimum order quantity on the price, and the price file the price came from.
- Rules: make a rule an error or a warning, or switch it off. A rule that is off is not checked and is not shown.
Copy From Old Completeness Settings appears when the site has the earlier completeness settings (the per-section field lists the product list's completeness filter uses). It fills the sections with the same fields, so the quality score counts what the earlier score counted. A name that matches a section goes into it; other names take the sections left over, renamed; the sections left unused are not scored. Fields that are not product or price fields are listed and left out. One difference remains: the earlier score read price fields only from prices with an end date still to come, while the Data Quality page reads them from any live price, including one with no end date. Count effectiveToDate in a section to score prices without an end date down.
Only the changes from eBiz's defaults are stored, so a site that changes nothing keeps the defaults as they are updated. Reset to eBiz Defaults clears the site's changes. A save with a problem, such as a level with no checks, is refused with the reason.
The figures do not change at once. The next build scores every product again under the new settings; until then the page shows the last build, read with the settings it was built under, and says the settings have changed.
Why does it change unexpectedly¶
- The score moved without anyone editing data. Prices count only while they are live, so a price reaching its end date lowers the Pricing section and can add No live price. Products also become Not updated in over 12 months as time passes.
- An edit does not show yet. The page shows the last build. Press Rebuild, or wait for the nightly build.
- The figures differ from the completeness filter on the product list. The product list still uses the older completeness percentage held in the search index, which counts a different set of fields.
- The figures moved after a change of settings. A build after the Data Quality settings change scores every product under the new ones, so scores, levels and rule counts can all move on that day's point of the trend.
- The trend chart is empty or short. It gains one point per day from the builds. Days with no build have no point.
What it is not¶
- It does not change any product data; it only reports on it.
- It is not Data Yard's own scoring. The levels are aligned with Data Yard's but are worked out from the fields eBiz holds.
- It does not check that values are true, only that they are given and well-formed.