Exposing "complex" JSON via RapidMiner Server

rfuentealba

Hello, World!

I am using RapidMiner Server to expose an API that provides zones and coordinates to a custom map. Each zone can contain at least three coordinates, and the custom map creates drawings on these coordinates and performs decorations with other information. I currently have two endpoints looking like this (the rest of the info has been dropped for clarity):

Zones:

[

{"id":1,"zone":"zone 1"},

{"id":2,"zone":"zone 2"}

]

Coordinates:

[

{"id":1, "zone_id": 1, "x": 0, "y": 0},

{"id":2, "zone_id": 1, "x": 1, "y": 1},

{"id":3, "zone_id": 1, "x": 2, "y": 2},

{"id":4, "zone_id": 2, "x": 10, "y": 10},

{"id":5, "zone_id": 2, "x": 10, "y": 20},

{"id":6, "zone_id": 2, "x": 20, "y": 20},

{"id":7, "zone_id": 2, "x": 20, "y": 10}

]

How feasible is to end up with this as a JSON output?

[

{"id":1,"zone":"zone 1","coords":[{"x":0,"y":0},{"x":1,"y":1},{"x":2,"y":2}]},

{"id":2,"zone":"zone 2","coords":[{"x":10,"y":10},{"x":10,"y":20},{"x":20,"y":20},{"x":20,"y":10}]}

]

The idea is to just embed the coordinates inside the result, as an Array, and expose it. Currently I expose this as two separate API endpoints, but the designer has to build the object properly using JavaScript on the client side (let alone creating two requests to the RapidMiner Server, which is less than ideal), and that scenario is far from ideal.

I don't have XML to provide this time, but imagine two simple "Retrieve" operators connected to the results window.

I would appreciate your help, Thanks in advance!

kayman

Not the most elegant way, but using concatenations, aggregations and some replacements gives you the right results.

Based on your example the below works (order is a bit different but in essence JSON doesn't care about that, and it is easy to fix if needed)

<列出关键= " jsonpath_queries " >


























<列出关键= " jsonpath_queries " >














< from_op = "重新连接place Tokens" from_port="document" to_op="Cut Document" to_port="document"/>























<参数键= " replace_what" value="^(.*)$"/>





<参数键= " replace_what" value="^(.*)$"/>































< from_op = "重新连接place" from_port="example set output" to_op="Replace (2)" to_port="example set input"/>
< from_op = "重新连接place (2)" from_port="example set output" to_op="Generate Concatenation" to_port="example set input"/>


< from_op = "重新连接name" from_port="example set output" to_op="Aggregate" to_port="example set input"/>

< from_op = "重新连接name (2)" from_port="example set output" to_port="out 1"/>


















< from_op = "重新连接place Tokens (2)" from_port="document" to_port="output 1"/>

How does it work :

First we get the JSON imports and convert them to proper datasets, next we concatenate x and y field while adding the variable names to the data itself. Then we aggragate all of the coordinates and produce a first json file. This does not contain the final data yet, but enough 'marks' that can be used to construct the remaining JSON logic with some basic find / replace logic.

As stated, not really high level and far from optimal, but it does the trick.

Good luck with it!

rfuentealba

Hello, World!

I have new data. This is the farthest I could go with my limited knowledge on how to operate with JSON. I still can't enclose the "Registration Subtypes" into an enclosing string or something that can help me achieve the desired results.

<参数键= " replace_what" value="Registration Subtype Label_"/>








< from_op = "重新连接trieve Registration Types" from_port="output" to_op="Join" to_port="left"/>
< from_op = "重新连接trieve Registration Subtypes" from_port="output" to_op="Pivot" to_port="example set input"/>

< from_op = "重新连接name by Replacing" from_port="example set output" to_op="Join" to_port="right"/>

Is there any kind of trick I can use to actually form the desired JSON file? I'm still researching my options, as long as these don't include using an external thing to build JSON

All the best,

Telcontar120

正如我们所讨论的其他线程,RapidMinerability to deal with complex JSON formats is somewhat limited. I am not aware of any way for you to get RapidMiner to create the JSON file with the nested array structure that you want. But I would love to see if any of the other experts in the community have a solution for this!

SGolbert

Hi Rodrigo,

can you share the json files or some process/script that generates the data?

It looks like the solution is a join on zone_id, but I am surely missing the point somewhere.

Regards,

Sebastian

sgenzer

Hi Rodrigo,

I battled this for about 45 min and could not get it. RapidMiner just does not have good JSON tools right now.Perhaps there are some good Python libraries that will do this trick for you?

Scott

rfuentealba

Hi@SGolbert,@sgenzerand@Telcontar120

Here is the result of my research:

@SGolbert: withjoining, if you have an example set with X categories and another with Y elements per category, you will end up with (XY) example sets. What I am looking for is something that help meincluding一个例子设置为在另一个exampl属性e set, and the inverse operations that I would callmappingandflattening. With that, you end up with X categories and an array (or hash) inside each category.

After reviewing tons of comments from the community (whoa, it is indeed massive!), googling around, and sitting down at the RapidMiner code (because I like my source like I like my mind: open! #noblackboxes to the core!) to see if that's feasible to be implemented, it seems that the tabular nature of a standard example set can't cope with the structural nature of JSON.

In the end, it is not (or it shouldn't be, at least) a massive amount of work to create anincludeoperator that does what I want, but I wonder if it is practical to implement other things, such as pre-flattening the object before training a model or applying it, and whether it should be considered (Creating such an object might help implementing SOAP Web Services from RapidMiner Server, which is a desired feature in some large companies and something that can take advantage from the real time scoring system, for example).

Thanks to everyone for their answers!

P.S.-@sgenzer, does it sound too mind-boggling to take this idea as a feature, a new plugin, or an idea for the RapidMiner Wisdom hackathon?

All the best,

sgenzer

@rfuentealbaput it this way - if the product team hears me yell "APIs" and "JSON" one more time, they will strangle me.

more seriously, our friend@Telcontar120has an idea open for voting that you are welcome to vote and comment on.https://community.www.kenlockard.com/t5/Product-Ideas/JSON-file-rotation/idi-p/49638

Scott

rfuentealba

Hi@kayman,

This looks awesome! Thank you. I haven't tried it extensively for my use case, but saw some things that might help me building the JSON extension I'm writing. Thanks again!

All the best,

Rodrigo.