Streaming Results Lazily in Laravel Query Builder: Tutorial & Examples

Introduction

Building efficient web applications often requires handling large datasets. In Laravel, one of the most powerful PHP frameworks for web development, querying large datasets can sometimes lead to memory issues and slow performance. This is where streaming results lazily using the Laravel Query Builder can be particularly advantageous. In this tutorial, we’ll explore how to leverage this approach to improve performance and resource utilization.

Understanding Lazy Collection

Laravel 6.0 introduced the concept of Lazy Collections, allowing us to work with very large datasets by leveraging PHP’s generators to keep memory usage low. A generator allows you to iterate over a set of data without needing to load everything into memory at once, which is perfect when dealing with a large number of records.

Scenario Setup

Imagine you have an application with a large table containing user logs that you want to process. For the sake of example, let’s pretend we need to analyze this huge dataset to extract some insights. Using a traditional Eloquent approach might lead to a memory exhausted error since Eloquent will try to load all records into memory. This is where streaming the results lazily can save us.

Setting up the Lazy Stream

$logs = DB::table('user_logs')->lazy();
$logs->each(function ($log) {
    // Process each log item
});

This snippet uses the lazy() method on the Query Builder which allows retrieving one Eloquent model at a time using PHP’s generator function. This method splits the result into smaller chunks and fetches them when needed.

Processing a CSV File

As an example, let’s say you want to process a CSV file without loading it fully into memory. Laravel’s Lazy Collection helps us to handle this task efficiently.

$path = storage_path('app/large-dataset.csv');
$csv = LazyCollection::make(function () use ($path) {
    $handle = fopen($path, 'rb');
    while (!feof($handle)) {
        yield fgetcsv($handle);
    }
    fclose($handle);
});

$csv->each(function ($line) {
    // Process each CSV line
});

In this code, we create a Lazy Collection instance by passing a closure to the make method. The closure uses PHP’s fgetcsv() function to read and yield one line at a time from the CSV file. Processing each line is done inside a each method call, which loops over the generator

Chunking Results

Laravel’s chunk method also provides a great way to process large datasets. However, the chunk method loads the complete chunk into memory, which can still be an issue with extremely large datasets or very limited memory constraints.

DB::table('user_logs')->orderBy('id')->chunk(100, function ($logs) {
    foreach ($logs as $log) {
        // Process each log in the chunk
    }
});

Leveraging Cursors for Memory Efficiency

In contrast with the chunk method, Laravel 7.x introduced cursors as a way to use PHP’s generators to iterate through database results without ever loading the full list of records into memory.

$logs = DB::table('user_logs')->orderBy('id')->cursor();

foreach ($logs as $log) {
    // Process each log
}

Use Case: Emailing Users

Imagine you want to email all of your users without overwhelming your server’s memory. Here’s a streamlined way to do it:

foreach (User::cursor() as $user) {
    Mail::send('email.view', compact('user'), function ($message) use ($user) {
        $message->to($user->email);
    });
}

With the cursor method, each $user object is only held in memory for the duration of the loop’s body and is disposed of afterward, significantly reducing memory load.

Best Practices and Considerations

• Use eager loading wisely: If you need related models, remember that lazy loading might cause the N+1 query problem. Instead, use with() to eager load relations as needed to keep things efficient.
• Database Cursors vs. Lazy Collection: Database cursors are best when dealing strictly with database results. Lazy Collections are broader and work well with any enumerable PHP structure.
• Testing and Optimizing: Even with lazy loading, always test to optimize your queries for better performance.
• Exceptions handling: Be aware of how exceptions are handled within your foreach loops, as this might affect the iteration process.

Conclusion

Using Laravel’s Lazy Collection and cursor features can greatly improve the performance of your Laravel applications when it comes to handling large datasets. By fetching and processing one row at a time, your application can conserve memory and reduce the likelihood of performance bottlenecks. Experiment with these options and find the right balance for your specific application needs and constraints.