Tag Archives: json

Generate huge JSON files with custom PHP >5.3 class

Lately, I’ve been working on transitioning XML feeds to JSON format on big video site.  We generate these feeds in order to feed external search service with results. It’s similar to sitemap, but it provides more detailed information about the pages.

This task is challenging because of the following problems that need to be resolved:

  1. The feed need to represent over 500 000 database entries i.e. videos. It’s just not possible to generate huge PHP multidimensional array with more than 500 000 elements and pass it to json_encode(). Obviously, you need to generate small JSON objects (chunks) concatenated with hand-coded strings and so build the full feed.
  2. The development and production servers we use are equipped with outdated PHP version 5.3.27. That means:
    – No meaningful error messages because json_last_error_msg() function it’s not available prior PHP 5.5
    – No JSON_PRETTY_PRINT, JSON_UNESCAPED_SLASHES, and JSON_UNESCAPED_UNICODE
  3. The code should be easy to test and maintain, so it should provide meaningful debug information and error handling.

Continue reading →