Moving a large store catalog without crashing it
A large Interspire store: a catalog export that kept crashing the site, rebuilt to run in batches, resume after failures and show progress; a custom filtered search; and a form too big for PHP, sent in chunks.
A long-running client had a large Interspire Shopping Cart store and needed its whole product catalog moved to another website. The obvious approach, exporting everything in one go, took the site down.
The problem
One request tried to read, transform and send every product at once. It ran out of memory and time, the server gave up, and the store went down with it. There was no record of how far it had got, so the only option was to start again and crash again.
What I did
Batches instead of one big job. I split the export into batches of a few hundred products. Each batch was small enough to finish well inside the server's limits, so the store stayed up while the export ran.
Resume instead of restart. Some products took much longer than others to process, and one slow batch could still time out and stop the run. I recorded the last product completed after every batch, and added retries: if a batch failed, it was tried again, and if the run stopped, it picked up from the last completed product instead of from zero. A failure cost one batch, not the whole export.
Show that it's working. A large export takes a while, and a screen that doesn't change looks broken, so people refresh it or close it, which is the worst thing they can do mid-run. I added a live counter of products processed and remaining, so whoever started it could see it moving and leave it alone.
A search that matched how customers shop
On the same store the client wanted search results filtered and ordered in a very specific way, which the platform's built-in search couldn't do. I wrote a custom query that combined all of their filter conditions and returned results in exactly the order they asked for. The client was very happy with the result.
The form that lost its own data
One form on the same store had so many fields that, when it was submitted, PHP quietly dropped part of it. There was no error: the extra fields simply never arrived. PHP caps how many fields a single request may carry (1,000 by default) and how large it can be, and this form went past those limits.
Raising server limits would only have moved the problem, so I changed how the form was sent. The user still clicked one button, but the browser now submitted the fields in chunks over AJAX, each one well within the limits, and the backend put them back together and saved them as one record. Every field arrived, every time.
What I took from it
Anything that touches a whole catalog should assume it will fail partway, and be built to continue rather than restart. And when something takes time, showing progress matters almost as much as being fast.