How to loop through all files in a directory and change macro values for each file?

asnani_sorathasnani_sorath MemberPosts:5Contributor I
edited December 2018 inHelp

Hi,
I am using Linear Regression Model to train a Model. I have several training and test files stored in different folders.

For example,
Training files are stored in:
D:/path/to/training/files/s2_merged_train.csv
D:/path/to/training/files/s3_merged_train.csv
D:/path/to/training/files/s5_merged_train.csv
....

Similarly the Testing files are stored as:
D:/path/to/testing/files/s2_merged_test.csv
D:/path/to/testing/files/s3_merged_test.csv
D:/path/to/testing/files/s5_merged_test.csv
.....
There are 271 training and 271 testing files. File names have the same pattern but they are not continuous, that is, there may be s2_merged_train.csv, s3_merged_train.csv but may not be s4_merged_train.csv. Similarly for the test files.
I have written the following process which is working correctly for given inputs. For file names and some attribute name, I am using macros, and passing the macro values through command line.
I am also storing the resulting file and the performance vectors in files.
I want to execute that process for all the training and testing files without inserting the macro values manually.
After searching on the internet I found that there is a Loop Files operator in RapidMiner which can be used to solve the problem, but I am having trouble in changing the macro values for each file.
My xml code is as follows:
















<参数键= " first_row_as_names " value = " true " / >




































<参数键= " first_row_as_names " value = " true " / >












<运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store" width="90" x="313" y="34">


































<运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store (2)" width="90" x="313" y="289">









































































<运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store (3)" width="90" x="1251" y="238">



For executing through command line, I am using the following command in Windows cmd:

C:/Softwares/RapidMiner/RapidMinerStudio/scripts>rapidminer-batch.bat "//Local Repository/processes/rp2" "-Mtraining-file=D:\ME_Thesis\Data_v2\experiments\x_prediction_time\10min_ahead\training\input_training_merged\s2_merged_train.csv" "-Mtraining-repository=//Local Repository/data/s2_train.csv" "-Mtraining-role-attribute-name=s2predicted" "-Mtesting-file=D:\ME_Thesis\Data_v2\experiments\x_prediction_time\10min_ahead\testing\input_testing_merged\s2_merged_test.csv" "-Mtesting-repository=//Local Repository/data/s2_test.csv" "-Mtesting-role-attribute-name=s2predicted" "-Mperformance-file=D:\ME_Thesis\Data_v2\experiments\x_prediction_time\10min_ahead\performance\s2_performance.res" "-Mresult-file=D:\ME_Thesis\Data_v2\experiments\x_prediction_time\10min_ahead\results\s2_result.csv" "-Mresult-repository=//Local Repository/data/s2_result.csv"

There may be 2 solutions for the mentioned problem.
1. Use RapidMiner operators to execute the process on all the available files
2. Write a batch file and include the the commands with parameters.
I don't have any prior experience in writing Windows Batch Files. So I would prefer the first solution.
Can anyone guide me how to use Loop operators and macros for this purpose?
Any help would be highly appreciated.
Thanks.

Tagged:

Answers

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, MemberPosts:1,761Unicorn

    @asnani_sorathYour XML file is not valid, please open the XML view and and cut and paste from there.

  • asnani_sorathasnani_sorath MemberPosts:5Contributor I

    I have copied the following code from .rmp file. I hope it would be correct now.





























    <参数键= " first_row_as_names " value = " true " / >
































    <参数键= " first_row_as_names " value = " true " / >










    <运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store" width="90" x="313" y="34">


























    <运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store (2)" width="90" x="313" y="289">



























































    <运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store (3)" width="90" x="1251" y="238">






    <连接from_op = "设置角色“from_port = "榜样output" to_op="Linear Regression" to_port="training set"/>


















  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, MemberPosts:1,761Unicorn

    Ok, that works. I think you want to use the Loop Files operator to load in your training and testing data sets. This way you can use pre-defined macros like %{file_name} and so forth.

    sgenzer
  • asnani_sorathasnani_sorath MemberPosts:5Contributor I

    Dear Thomas, can you kindly provide an example of using that?
    Should I use 2 Loop Files operators? 1 for training files and another for testing files?
    在我的每个文件,标签属性是names as: 's2predicted' in files s2_merged_train.csv. and s2_merged_test-csv; 's3predicted' in file 's3_merged_train.csv' and 's3_merged_test.csv' and so on.
    How to use macros for changing these values according to each file?


    Thanks.

  • asnani_sorathasnani_sorath MemberPosts:5Contributor I

    Is there no one to help? I am constantly searching and trying but no results.

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, MemberPosts:1,761Unicorn

    @asnani_sorathnext time use the '@' symbol to get someone's attention, we're all busy with work and spend our time on the Community freely. Response time varies.

    W.R.T to your process, I would do something like this. In addition you need to define your macros like %{performance} first before you can use them. That's another reason why things break.


























    <运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store" width="90" x="313" y="34">

    Path with ${file_name}









    <连接from_op = "设置角色“from_port = "榜样output" to_port="output 1"/>























    <运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store (2)" width="90" x="380" y="238">































    <运营商激活= " true "类= compatib“商店”ility="8.1.000" expanded="true" height="68" name="Store (3)" width="90" x="1117" y="289">



























    sgenzer
  • asnani_sorathasnani_sorath MemberPosts:5Contributor I

    Dear@Thomas_Ott, thank you very much for your response.
    Your suggested process didn't work for me because the Linear Regression operator is out of the loop, so it won't execute on all files.
    Never mind, I have solved the problem by generating a file with all the possible parameter values, and I have used 'Loop Examples' operator instead of 'Loop Files'. Now it is working perfectly fine.
    Many thanks for your precious time.
    Regards,
    Sorath Asnani

    sgenzer Thomas_Ott
Sign InorRegisterto comment.