Load structured data (XML, JSON, CSV, TXT) from files or network (HTTP, HTTPS)

Feature Requests
, , ,
21

Just playing with the new 1.2 A4 vuo.data.fetch, great job guys!

One question for loading structured data: will we have the ability to use Keys to access data? And if so can the data be output in correct Vuo parlance? (if formatted correctly)?

XML Such As:

<myxml>
<foo>1</foo>
      <bar>
          <x>2</x>
          <y>3</y>
     </bar> 
</myxml>

Would output if using a “Vuo.list.get.with.key” node (yet to be created):

  • KEY: foo = 1
  • KEY: bar = “x”:2,“y”:3

This way one can use Index & Keys to control certain aspects of complex compositions. (I already do this for controlling compositions in QC- so I really would love to be able to rebuild my work in Vuo!)

In the above example, bar could be the key for the x,y position of a layer, so then I could add multiple values for bar- and simply move through them via indexes! If an index does not contain a ‘bar’ key then ‘Vuo.list.get.with.key’ wouldn’t spit out anything.

UPDATE: I think this is covered by the request “Dictionary Support”, however I believe it may have low votes due to assumption that this feature request would encapsulate the other? I don’t know how useful a list generated from XML would be without its dictionary.

22

Chosen to be implemented.

23

Does this include saving data as well?

24

Thanks all for your feedback. Update on the plan — it includes:

  • Table-style (CSV, TSV) and tree-style (JSON, XML, HTML) data
  • Higher-level types (than lists or dictionaries) for efficiently working with data
    • Table with ability to get row/column by header or index, ability to add/remove/change row/column
    • Tree with ability to search by XPath/name/attribute/content, extract attributes/content, build tree hierarchically
  • Save Data node to save to file
  • Converting tables and trees to/from the Vuo Data type for use with the Save Data and Fetch Data (read from file) nodes.

New, related feature request: Advanced HTTP(S) query nodes

25

As list-in-list will no longer be developed in 1.3- can this node be easily updated from a 1D list to "n"D list?

26

@alexmitchellmus, I’m not sure what you mean. Maybe it would help if you gave an example / describe the task you hope to accomplish?

27

@jstrecker. In xml you can have an item of item:

<red><blue><green>"1"</green></blue></red>

So green 1 belongs to blue and blue belongs to red.

If you read out red you get a list of blue, which has a list of green which is value 1.

So my question here is if XML is developed in 1.3- then won’t team Vuo need to re-code aspects of it to work with list in lists?

This is to say the above list would not work with 1.3 XML reader as its not a 1D structure? If I am wrong let me know.

28

This is the first 3 cues from a show I worked on (Wonder if anyone can guess the show?!). Here you can see the main XML element is data, then each other element can have its own elements. In QC I can use this method to cue animation, or switch off parts of the composition. Its very powerful as all I need to do is edit the xml file- and my whole show is there. I also don’t have to worry about cue numbers, as if I want to insert a cue I simply do. Yet if I wanted to I could simply call the scenes cue numbers anyway. Does this explain a bit more @jstrecker?

<data>

<blackout>
</blackout>

<sunrisePrideRock>

<prideRockIn>1</prideRockIn>

<playTo>100</playTo>

</sunrisePrideRock>

<blackout>
</blackout>

</data>

29

This is awesome that XML is going to be supported this next update! Great news! I am so looking forward to moving a project into VUO from QC and this is going to be a huge step forward in doing that. Thank-you!

30

@alexmitchellmus, with this feature request, a composition to parse the “pride rock” example you gave would work something like this:

  • Fetch Data -> Parse Tree
    • output data type: Tree
  • Find Tree Item with Name
    • inputs a Tree and an XML tag name (e.g. “playTo”)
    • outputs a list of matching children, as Trees
  • Get Tree Item Content
    • inputs a Tree (e.g. the match found above)
    • outputs the content (e.g. “100”)

Instead of multidimensional lists, you’d be dealing with the Tree data type. (That’s what I meant by “higher-level types” in my comment above.)

1 Like
31

I’m really excited for the ability to parse CSV files. I’m working on a number of video tools to generate special effects. and CSV tables are ideal for structuring the actions to be taken to multiple input streams of video, Each line of the cvs table specifies the actions for the input streams.

32

@jstrecker if XML parser looks at the xml like a tree structure, can we also output a tree from an index? Such as asking the parser for XML item number 3? Etc?

That is how I would need to use the XML as well, moving though the elements as both index and dictionary support.

So that means I could find tree item with index number.  

33

@alexmitchellmus, yes, we plan to support accessing tree items by index. You’ll be able to feed a tree (or sub-tree) into a node that outputs a list of the tree’s children, which you can index into with Get Item from List.

34

I need this too, so happy it’s on it’s way. Where do we download alpha/betas?

35

Do we know what release version this feature has been scheduled for? The project I need it for isn’t until early June 2017 but I wanted to make sure I had enough lead time to test and ensure my composition will be working smoothly first.

I found myself in a very awkward spot if I stay with QC for this year because the composition I am running not only needs a computer with a very specific set of ports but also one able to run OS X 10.7 for QC feature compatibility which leaves me a very short list of Macs this composition can even run on. My 2011 17" MacBook Pro is dying a slow and painful death and so I am hoping to switch over to VUO which makes my list of possible compatible Macs much bigger. Here’s hoping it’s coming soon.

36

@cwilms-loyalist, it’s scheduled for Vuo 1.2.6.

@useful_design, it’s been a while since we’ve done an alpha/beta release, but they are on https://vuo.org/releases .

1 Like
37

Somehow I missed your response but am SO glad to hear it’s coming in Vuo 1.2.6… if you need any alpha/beta testers I can do a little testing with my planned project. :)

My initial need will just be to have my project read a datamatrix code with a camera, extract a 4-digit code from the delineated text string and then use that 4-digit code to find the appropriate replacement text in an XML file for displaying on screen.

38

@cwilms-loyalist and anyone else who’d like to be testers — we just published a Vuo 1.2.6 alpha release that includes XML/JSON nodes. We’d appreciate if you could try them out and let us know if they do what you need them to do.

The final 1.2.6 release will include much more stuff, including CSV nodes, but we wanted to get your feedback on the XML/JSON nodes sooner.

If you find any problems with the alpha release, please post a bug report. If you have questions/suggestions about the scope of this feature request (what the new nodes will/won’t be able to do), please comment here. For other questions/suggestions, probably best to a new discussion thread so it’s easier to follow the conversation.

3 Likes
39

Great!!! I’ll try it out as soon as I can! I’ll try and set some time aside tomorrow to do that. I’m away next week so I’ll really try and squeeze it in. This is amazing!

1 Like
40

I’ll have some time over the next week (after election campaigning on Climate action) to check it out. I have a few interesting datasets I’ve been wanting to get into Vuo. I tend to use CSV out of spreadsheets but it’s easy enough to use GREP to XMLify them.