What's with the 1,000 product import limit?
- It's a Zazzle thing
- Why does this affect the importer?
- Why does Zazzle have this limit?
- Pushing the limit
It's a Zazzle thing
If you have a large store you may have noticed that the initial import only brought in the first 1,000 (1k) products. This is not a limitation of this website. It is actually a Zazzle limitation.
For technical reasons, Zazzle's developers chose to put a cap on how many results you can see returned from any search. Try a search for popcorn for an example. You should see it indicating more than 10k search results:
But if you page your way through the search results as far as you can you'll see there are only 1k shown. To illustrate this, add "ps=100" into the URL. "ps" is short for "page size". And then "pg=9" means "page 9".
See how there is no page 10 listed? Page 9 is showing you products 901 - 1,000. There is nothing more.
That's the 1k brick wall you're hitting.
Why does this affect the importer?
This 1k limit affects the Zazzle RSS feeds just the same as the human-friendly search pages. The storewide product importer relies on Zazzle's RSS feed to discover products in a store. If it could see past the first 1,000 products then it could import them as well.
Why does Zazzle have this limit?
I can only speculate as an outsider. One technical reason may be just to make searches perform better. One reason may be to prevent outside hackers from burdening their servers by requesting unreasonably large result sets. As a programmer I can tell you that these are unsatisfying answers because they do other things that prevent these from causing real strains on their system.
I can also speculate about business reasons for this limitation. The 1k limit makes it harder for outside systems to collect vast amounts of data from Zazzle. Imagine writing a simple webcrawler that can gradually download Zazzle's vast catalog of millions of products in 128-item search results over a few hours of hammering their RSS feed.
There is a business case for believing making it this easy to export their entire catalog would be bad for Zazzle. In fact I can quickly think of a dozen arguments for how this could hurt Zazzle. One of the most basic is that someone could set up a mirror site and draw away Zazzle's customers from their ability to effectively market in their own way. Another is that it would be easy to generate basic aggregate statistics with which to measure Zazzle against other companies. Zazzle probably considers this important competitive information that they don't want easily discovered.
The bottom line though is that there is this 1k limit on just about every page you can think of on Zazzle that lists things. It's there and it presents technical challenges for us to deal with. That's our problem.
Pushing the limit
The newer version of the product importer does go beyond the 1k limit using a clever trick. Originally I was just getting the first 1k of products from your store sorted by recency (newest first). Now I add to that the results of sorting by oldest first, which can get me up to 2k products. I also then sort by lowest price and then highest price, getting me up to 4k products total. And then I sort by most and least popular, getting me up to 6k. It's not guaranteed to be 6k of course. There's bound to be a lot of overlap in the 6 different sort orders. But it's likely to get a large fraction of most stores that are over 1k.