moving average calculation for activejob rails

Introduction

Oh, well that was a mouthful of statement for a cover title!

Anyway, let me break it down in simpler terms. In a nutshell, what I want to show you, is a way to show the end users when a process is about to finish using a live update with turbo stream. Similar tutorials, I believe, have been written to show live updates of a progress bar. So, here’s another one.

Problem Statement

The problem I had, which you can modify to suit your own needs: I needed to delete thousands of users from a backend rails application using a cvs file. Then I needed to show the admin end user the progress in %, the progress bar being updated and the estimated time remaining. All that in real time.

Time remaining calculation

Well, in simple math, if you have thousands of records to be processed and you know the average time of processing a single record, then the time remaining or the estimated time remaining would be:

time remaining = average time to process a single record * (how many records you have - the index number of current record number you are processing)

So, if you have 4000 records and you know that it takes about 0.5 secs average time to delete a record then if you are working on record 3400, your time remaining would be

time remaining = 0.5 secs \ (4000 - 3400) = 300 secs or 300 sec \ 1 m/60 sec = 5 minutes

Average time of a record’s deletion

In order to calculate the average time, we use a math method called simple moving average calculation. For my case, I created a ring buffer which contains numbers; each number representing the time every single record of the last 40 records took to get deleted. The number “40” was arbitrary. I could’ve created a ring buffer of size 10 or 20 but I chose 40 in order to smooth sudden spikes in average.

You can find here more about simple moving average: https://www.investopedia.com/terms/s/sma.asp

I used the following to implement a ring buffer. Actually, I stole the code from

https://github.com/celluloid/celluloid/blob/master/lib/celluloid/logging/ring_buffer.rb

where I added the following method to implement average calculation:

# calculates the mean average of all the values in the buffer provided that
    # the buffer is already full
    def average
      if self.full?
        @buffer.inject{ |sum, el| sum + el }.to_f / @size
      end
    end

Here’s is the whole file: https://gist.github.com/nkokkos/10f30e93101412b70782c7d4b9c43e39

Background job and turbo stream

I used an ActiveJob to loop through the csv file, calculate the time it took for each user to get deleted and calculate the time remaining. Deleting the user was a blocking process so we had to wait until deletion was complete. The following code is pretty much self explanatory. I used the example here https://makandracards.com/alexander-m/53156-measure-elapsed-time-the-right-way to measure the elapsed time correctly. Then I used the Turbo::StreamsChannel.broadcast method to broadcast the live data to view.

ring = RingBuffer(40); 
line_counter = 1;
CSV.foreach(file_path, headers: true, col_sep: ",") do |row|
   time_start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
   # calculate elapsed time the correct way:
   # https://makandracards.com/alexander-m/53156-measure-elapsed-time-the-right-way
   # time_start = Time.now
   # time_start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
   # do the work.....
   # time_finish = Process.clock_gettime(Process::CLOCK_MONOTONIC) - time_start
   result = connection.delete_user(row[0].to_s)
   #job_logger.debug result.to_s
   time_taken =  Process.clock_gettime(Process::CLOCK_MONOTONIC) - time_start
   ring << time_taken
   percent = ((line_counter / file_lines) * 100).round(2)
   time_left = ring.average * (file_lines - line_counter)
   eta = "#{time_left.round(2)} secs or #{(time_left/60.0).round(2)} minutes"
   Turbo::StreamsChannel.broadcast_replace_to("progress_bar",
     target: "counter",
     partial: "bulk_actions/counter",
     locals: { counter: percent.round(2), student: row[0], eta: eta})
   line_counter = line_counter + 1.0
 end

In view, we subscribe to a turbo stream tag “progress_bar” and we render the partial “bulk_actions/counter.html.erb”

 <%= turbo_stream_from "progress_bar" %>
 <%= render "bulk_actions/counter", counter: 0, student: "", eta: "" %>

In partial “_counter.html.erb”, we are constantly updating the values from turbo stream:

<%= turbo_frame_tag "counter" do %>
  <p>Percent done: <%= counter %>%</p>
  <p>Current user: <%= student %></p>
  <p>Time to completion: <%= eta %></p>
  <div class="progress mb-3">
    <div class="progress-bar progress-bar-striped" role="progressbar" style="width: <%= counter %>%;" aria-valuenow="<%= counter %>" aria-valuemin="0" aria-valuemax="100"><%= counter %>%</div>
  </div>
<% end %>

In the end, our bootstrap progress bar being updated in real time:

Conclusion

Giving feedback to the user in real time is of the utmost importance. Besides this, it was really fun implementing this solution using turbo stream updates and background job. I hope you find it useful.

Use a moving average calculation to predict estimated time remaining while updating progress bar with turbo stream.

Table of contents