Accessing Wikipedia API using RapidMiner Web Mining Extension

sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM ModeratorPosts:2,959Community Manager
edited February 2020 inKnowledge Base

This is a quick article about how to use the Enrich Data via Webservice operator (found in the Web Mining extension) to get information about Wikipedia via their REST API webservice. This API can find many different sources of information such as page views, formula grabs, unique device counts, etc.. Full documentation can be found here:https://wikimedia.org/api/rest_v1

This particular API is VERY easy to use - there is no authentication and the only limitation is a 200 query count per day. Simply enter the URL, insert the relevant attributes or macros, and set up the JSON paths to organize the output. Boom.

This is an example of a short process that check the page count of theRapidMiner Wikipedia page(of course) the day prior to when the process is executed.

























< =“query_type”价值=参数的关键"JsonPath"/>















https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/all-agents/RapidMiner/daily/&lt;%startdate%&gt;/&lt;%enddate%&gt;"/>











Enjoy!

Scott

Tagged:
JessForbesRM CraigBostonUSA yyhuang sharmar6 Pavithra_Rao

Comments

  • sharmar6sharmar6 MemberPosts:19Maven

    Thanks Scott..!!!

    :smileyvery-happy:

  • sharmar6sharmar6 MemberPosts:19Maven

    Hi Scott,

    I am using this API but couldnt figure out how to pass token and then again access a different endpoint.

    Could you please advice where to put the token and the query parameter.

    Thanks.

    Capture.JPG

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM ModeratorPosts:2,959Community Manager

    so the -H curl request means that it must be included in the header. With the Enrich Data via Webservice operator, it's done in the advanced parameter "request properties":

    Screen Shot 2017-09-21 at 7.03.56 PM.png

    API,看起来他们会让你包括the token right in the URL instead of in a header. So in the operator, you are just going to do the URL like it says:

    Screen Shot 2017-09-21 at 7.05.49 PM.png

    Same difference.

    Scott

  • sharmar6sharmar6 MemberPosts:19Maven

    Thanks.

    I also need to fetch a json response which is on another page, after authentication

    How do I make RM to use GET/other_data after authentication.

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM ModeratorPosts:2,959Community Manager

    so that's what this operator does: a GET request which generally returns a JSON file as a response. If you set the query type to Regular Expression and the query expression to .*, you will see the whole response.


    Scott

  • sharmar6sharmar6 MemberPosts:19Maven

    The response that comes after authentication is of no use. I need the response from GET/posts which has information I am looking for. But if I put the token in the url, it stops after giving the initial resonse (landing page). I need to access other page after getting authenticated. How should I form my url and request parameter so that I am able to authenticate and move to other page as well.

  • kludikovskykludikovsky MemberPosts:30Maven

    Check this BUG-Report!

Sign InorRegisterto comment.