chris-halcrow
Posts: 3
Joined: Mon Oct 29, 2018 6:24 am

Best approach for high volume real-time data

Mon Oct 29, 2018 10:40 pm

We're currently using Highcharts to display a high volume of data in a real-time line chart (a time series chart), and it's causing issues with the Chrome browser tab crashing after a relatively short period of time displaying data (about 20 mins). My question is what approach would be be best to use to ensure that the chart performs well and is reliable?

We poll data from various technical instruments. We then use a Highchart that displays new data every second, for a number of instruments, and the chart can span a period of up to a maximum 24 hrs. After about 20 minutes, the browser tab in Chrome that's displaying the line chart hangs. There are no error messages, although at some stages we have seen the message "Error 'Paused before potential out-of-memory-crash'". It seems to be due to the CPU maxing out, and memory use of the browser tab is continually increasing over the timescale of the time series (presumably until a maximum is reached, as we discard 'expired' data that's now beyond the current timescale of the chart).

The way we're implementing the real-time chart is by continuously updating the options I've tried a couple of different approaches to optimise the real-time chart however that hasn't fixed the problem. The way we update the x-axis is like this (passing in the minimum and maximum values for the current time period to be displayed on the x-axis, so that the chart is always animating to show a moving time window). For example, the chart may continuously display a current 24hr window up to the present moment:

chartObject.xAxis[0].setExtremes( ... )

The way we set the chart data is by initialising the chart with a buffered set of data and using something like this to set the data for each series:

chartObject.series.setData(data, false);

And then when new data is received, we delete any expired data (data readings for a time that's below the chart minimum), and append new data like this:

Append: this.chartObject.series.addPoint(latestDataPoint, false);
Remove: this.chartObject.series.removePoint(0, false);

This way this is being implemented, we are successfully appending and removing data (the data size isn't growing). Everything else in our implementation is optimised (there are no expensive redundant loops or anything else that's sub-optimal). For a chart that displays a 24hr time period with new data every second, the browser tab in Chrome fails after about 20 minutes.

Should we be implementing the chart as described or do we need a different approach for displaying this volume of data? Will the boost module guarantee that the chart performs reliably? Or do we need to use a Highstock chart instead of a Highchart? This is taking up a lot of time and we need an approach that will guarantee a successful outcome.

rafalS
Posts: 693
Joined: Thu Jun 14, 2018 11:40 am

Re: Best approach for high volume real-time data

Tue Oct 30, 2018 10:34 am

Hi, chris-halcrow!

You don't have to append and remove points using 2 separate methods. You can read in docs that in addPoint() method you can set shift argument to true. While adding a new one, every last data point will be removed and it shouldn't affect memory.
https://api.highcharts.com/class-refere ... s#addPoint

Could you test it and check whether it's still crashing?

Best regards!
Rafal Sebestjanski,
Highcharts Developer

chris-halcrow
Posts: 3
Joined: Mon Oct 29, 2018 6:24 am

Re: Best approach for high volume real-time data

Tue Oct 30, 2018 11:36 pm

Thanks Sebestjanski

I'm aware of the 'shift' argument, however for the data we're using, it's possible for the user to change the data polling rate. This means that if the user starts polling data more quickly, the 'shift' argument will then remove data points from the beginning of the dataset that have a longer time interval between points than the new points being added, so the chart will initally appear to 'recede' from the beginning of the time series. That's why I remove point manually, by removing only the points that have expired from the timescale currently being presented on the chart. Also, sometimes the initial data that's pre-loaded for the chart doesn't cover the entire timeseries, so when data is added, data at the start shouldn't be removed every time a new point is added (until it's old enough to expire off the chart)

I'll let you know the results of using 'shift' anyway. If it works, maybe I can find a way to shift data if we can see that the oldest point has now expired, but not if the point hasn't expired.

chris-halcrow
Posts: 3
Joined: Mon Oct 29, 2018 6:24 am

Re: Best approach for high volume real-time data

Tue Oct 30, 2018 11:40 pm

Hi Sebestjanski - using 'shift' didn't work anyway. We're displaying 6 series at a time, with a new data point added for each series ever second. That means there will be 7200 data points after 20 mins. The most I've been able to display is maybe around 20,000 - 25,000 points before crashing. So we don't need to use the boost module or a Highstock chart instead of a standard Highchart? This is the method we use to process data onto the chart (now using 'shift' when using 'addPoint':

Code: Select all

  processChartData(chartData: CwChartData) {

    if(chartData.series[0].dataPoints.length > 1) {
      // This is the initial dataset for chart initialisation
      
      const dataSeries: any[] = [];
      
      chartData.series.forEach(series => {
        series.dataPoints.forEach(dataPoint => {
          dataSeries.push([dataPoint.x, dataPoint.y]);
        });
      });

      for (let i = 0; i < dataSeries.length; i++) {
        if (this.chartObject.series[i]) {
          this.chartObject.series[i].setData(dataSeries, false);
        }
      }      
    }
    else {
      // This is a dataset of the latest values for each series - use them to update the data
      for (let i = 0; i < chartData.series.length; i++) {
        if (this.chartObject.series[i]) {
          const latestSeriesValue = chartData.series[i].dataPoints[0];
          let latestDataPoint : [number, number] = [latestSeriesValue.x, latestSeriesValue.y];
          this.chartObject.series[i].addPoint(latestDataPoint, false, true);
        }
      } 
    }
  }

rafalS
Posts: 693
Joined: Thu Jun 14, 2018 11:40 am

Re: Best approach for high volume real-time data

Wed Oct 31, 2018 12:37 pm

chris-halcrow,

It's hard to say what is causing your problem without an access to your app. Are you able to imitate your chart online (jsFiddle)?

There should be no difference between Highcharts and Highstock with dealing and adding a large number of points, but maybe boost module could help. The question is whether there is a problem with a large number of points or with a large number of operations you are trying to do on the chart every each second. To tell more, I would have to see the whole code of your chart.

Of course, you can try to use the boost module and see whether it helps. Then we would know whether the problem is related to the number of points.

Kind regards
Rafal Sebestjanski,
Highcharts Developer

marrocmau
Posts: 12
Joined: Mon Sep 10, 2018 8:25 am

Re: Best approach for high volume real-time data

Wed Dec 05, 2018 2:40 pm

I have the same issue.
I have 20 series and add 1 point every minute that means in 20 minute:t 20*20= 400 points will be added in one chart.
And in same page we have 6 or 8 chart (like you see in the image).

How can to resolve this "out-of-memory" issue?
Screen Shot 2018-12-05 at 15.38.18.png
Screen Shot 2018-12-05 at 15.38.18.png (137.45 KiB) Viewed 54 times

rafalS
Posts: 693
Joined: Thu Jun 14, 2018 11:40 am

Re: Best approach for high volume real-time data

Wed Dec 05, 2018 3:49 pm

marrocmau,

Have you tried addPoint() method with shift argument that I suggested above?

Here you can take a look at how exactly does it work: https://jsfiddle.net/BlackLabel/evypfr1L/
When the number of points is high, you add new point removing the last point in the same time.

Best regards!
Rafal Sebestjanski,
Highcharts Developer

marrocmau
Posts: 12
Joined: Mon Sep 10, 2018 8:25 am

Re: Best approach for high volume real-time data

Wed Dec 05, 2018 10:38 pm

rafalS wrote:
Wed Dec 05, 2018 3:49 pm
marrocmau,

Have you tried addPoint() method with shift argument that I suggested above?

Here you can take a look at how exactly does it work: https://jsfiddle.net/BlackLabel/evypfr1L/
When the number of points is high, you add new point removing the last point in the same time.

Best regards!
Yes of course.

For example with 3 chart in the page (see the image)
Screen Shot 2018-12-05 at 23.31.55.png
Screen Shot 2018-12-05 at 23.31.55.png (232.72 KiB) Viewed 44 times
after 45minutes this is the memory that iis used by page:
Screen Shot 2018-12-05 at 23.30.17.png
Screen Shot 2018-12-05 at 23.30.17.png (92.73 KiB) Viewed 44 times
The page is very slow and how "Memory footprint" will arrive at 2GB will crash.

rafalS
Posts: 693
Joined: Thu Jun 14, 2018 11:40 am

Re: Best approach for high volume real-time data

Thu Dec 06, 2018 12:04 pm

Well, I don't see any other reasons why your browser crashes. Could you send me your one simplified chart in jsFiddle? I will create a simple website with 10 same charts and dynamically loaded data and see if my browser crashes too.

Best regards
Rafal Sebestjanski,
Highcharts Developer

Return to “Highcharts Usage”