{"id":3242,"date":"2018-07-06T23:00:00","date_gmt":"2018-07-06T23:00:00","guid":{"rendered":"https:\/\/notebooks.dataone.org\/?p=3242"},"modified":"2019-05-24T17:47:21","modified_gmt":"2019-05-24T17:47:21","slug":"week-6-parser-metadata-mapper-using-apache-tika","status":"publish","type":"post","link":"https:\/\/notebooks.dataone.org\/extending-libmagic\/week-6-parser-metadata-mapper-using-apache-tika\/","title":{"rendered":"Week 6: Parser, Metadata Mapper Using Apache Tika"},"content":{"rendered":"

Hi All,<\/p>\n

This blog is in follow-up with my earlier blogs for the Project 4: Extending Libmagic for Identification of Science Resources<\/a>. After resetting our goals for rest of the project in the previous week. The goal is to extract metadata from different file formats using Apache Tika.<\/p>\n

Since we want to extract the metadata field from a given input file format, a custom parser for the file format needs to be created. Every file format has a different XML structure and different metadata field. A single parser cannot extract all the metadata fields present. using StaX class in Java, we created a parser last week for parsing the contents of the XML file, which worked fine. However, the StaX class provides functionality for the forward reading of the XML. We can’t query or extract specific XML tags or fields from it. The DOM class in Java provides us the flexibility for getting the fields using XPath queries or using the method getElementsByTagName.<\/p>\n

We created a parser and.a class XPathMetdata, which consist of XPath queries for all the metadata fields present in EML file as below: The XPath for the field is then passed to the parser method “getNodeValues”, which returns the List of the Nodes from the file.<\/p>\n

emlXpath.add(“\/\/dataset\/creator\/*”);<\/li>\n

emlXpath.add(“\/\/dataset\/creator\/address\/*”);<\/li>\n<\/ul>\n
This works fine, but if the XML file structure is changed or a field is added or removed, then it would fail. This approach won’t scale as well. We want to get specific metadata fields from the input file, hence a better approach is to get the metadata and its child nodes using the specific tags. The new method getMetadata uses the file and tags as input parameter and prints the node Name and node values for the metadata field. it also takes care of the childNodes for the passed XML tag.<\/p>\n
The next step in achieving our goal is to create the metadata fields and the mapper. Though we created it earlier but re-designing it will be a good thing. This is needed as the metadata fields are different across file format for example, <abstract> field in EML format has a child Node, whereas for onedcx it doesn’t. Same is for the other fields such as spatial, temporal etc. For now, the fields like title, address etc are mapped in the Mapper class.<\/p>\n
- metadataFields.add(“address”);<\/li>\n<\/ul>\n
  The metadadata fields present in the List are passed as the input to parser class and it returns the metadata field and values. The output is correct but still needs certain cases to be handled such as returning of the empty tags. In the coming week, our goal is to redesign the mapper and create the metadata fields for it.<\/p>\n
  That\u2019s all for now, see you all next week!<\/p>\n
  Have a great weekend!<\/p>\n
  <\/p>\n
  Resource links: Github-file_identification<\/a>, Github-DataONE Parser,<\/a> Project Plan<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
  Hi All, This blog is in follow-up with my earlier blogs for the Project 4: Extending Libmagic for Identification of Science Resources. After resetting our goals for rest of the project in the previous week. The goal is to extract metadata from different file formats using Apache Tika. Since we want Continue reading Week 6: Parser, Metadata Mapper Using Apache Tika<\/span>→<\/span><\/a><\/p>\n","protected":false},"author":113,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[389],"tags":[],"_links":{"self":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts\/3242"}],"collection":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/users\/113"}],"replies":[{"embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/comments?post=3242"}],"version-history":[{"count":4,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts\/3242\/revisions"}],"predecessor-version":[{"id":3347,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts\/3242\/revisions\/3347"}],"wp:attachment":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/media?parent=3242"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/categories?post=3242"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/tags?post=3242"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}