Fetch Data takes long time to load large lists

Steps causing the bug to occur

  1. Use Fetch Data for CSV list
  2. Compare load time of small, v’s large lists

Have you found a workaround?

Wait for list to load? One of my lists takes 45seconds to load 14290 items. Please find compositions attached.

Other notes

  • Vuo version: 1.2.1
  • macOS version: OS X 10.9
  • How severely does this bug affect you? It’s annoying but I can work around it.

Data Set Load Time.zip (326 KB)

DataSet Large (14290 items-43sec Wait).png

DataSet Small (220 items-instant load)).png

After checking this out more, this is indeed the same issues as: https://community.vuo.org/t/-/5326

However my data is quite huge (2MB) so maybe the fix wouldn’t be the same? Would the new “Structured Data” node set be more suited to this job? (Or a custom data loader which only outputs specific chunks of CSV data?)

I have done some testing, and although it doesn’t have to do with this node, I remade a text file reader node that automatically takes the data and chops it up using strtok(). I have attached the node & data. This allows data to be read with no noticeable lag at all. I created a CSV that was 17 MB and 114,320 and it loaded as the composition ran.

The issue with ‘split text’ must be huge because Vuo is so fast!

If anyone want to use the node its attached. I needed to make this to read in data values from a 3 axis sensor placed on an actor. Each time I was working on the project it would take 5 mins to load the data. Now instant. Any issues source is attached.

split_text_file.zip (3.83 KB)

Screen Shot 2016-02-23 at 11.43.47 pm.png

1 Like

@alexmitchellmus nice job !

Ran some tests, about 1 minute between the two. Blue is the built-in, magenta yours.

Test made with a 130 k CSV file of 6000 values

@jstrecker, is it ok to use strtok() in a node - or should I change it too strtok_r()? I thought vuo keeps nodes executing on each thread, only multi threading different nodes?

So it needs to be strtok_r() otherwise more than 2 nodes corrupts the other ones list! :-(

Fix attached. Try to do this with strtok()! :-D

split_text_file_reentrant.zip (3.95 KB)

1 Like

After checking this out more, this is indeed the same issues as: Split Text node becomes slow with long text

Confirmed. Since the Split Text bug is already fixed in our dev code, we tested your composition and CSV with it.

Hey @jstrecker is the new code in vuo as fast as the example I posted using strtok_r? I really do need fast data reading! :-) 1.3 looks like its developing nicely! Great Job can’t wait!!!

The new code parses your CSV file instantly (human perception; haven’t timed it).

For Load structured data (XML, JSON, CSV, TXT) from files or network (HTTP, HTTPS) we’ll be adding a node for parsing CSV that supports double-quote escaping, which Split Text does not.

Absolutely Fantastic! :-)