{"id":3183,"date":"2018-06-08T20:31:25","date_gmt":"2018-06-08T20:31:25","guid":{"rendered":"https:\/\/notebooks.dataone.org\/?p=3183"},"modified":"2019-05-24T17:47:39","modified_gmt":"2019-05-24T17:47:39","slug":"week-2-created-tested-and-committed-magic-file-with-the-libmagic-library","status":"publish","type":"post","link":"https:\/\/notebooks.dataone.org\/extending-libmagic\/week-2-created-tested-and-committed-magic-file-with-the-libmagic-library\/","title":{"rendered":"Week 2: Created, tested and Committed magic file with the Libmagic library."},"content":{"rendered":"<p>Hi All,<\/p>\n<p>This is the second week of my internship, and&nbsp;below are the tasks that were completed during this week.<\/p>\n<ul>\n<li><strong>Adding the patterns for the rest of the file formats into the dataone magic file.<\/strong><\/li>\n<\/ul>\n<p>Continuing the work from the last week, we were able to create additional patterns for the detecting the file format using the file command with the custom magic file. Below are the 13 file formats that we wanted to identify. We created different files for each of the format initially but combined them later into one by adding additional tests level. The custom &nbsp;magic file can identify the 12 file formats successfully out of 13.<\/p>\n<ol>\n<li>http:\/\/ns.dataone.org\/metadata\/schema\/onedcx\/v1.0<\/li>\n<li>http:\/\/datadryad.org\/profile\/v3.1<\/li>\n<li>FGDC-STD-001-1998<\/li>\n<li><strong>FGDC-STD-001.1999<\/strong><\/li>\n<li>http:\/\/purl.org\/ornl\/schema\/mercury\/terms\/v1.0<\/li>\n<li>http:\/\/www.isotc211.org\/2005\/gmd<\/li>\n<li>http:\/\/www.isotc211.org\/2005\/gmd-pangaea<\/li>\n<li>http:\/\/www.isotc211.org\/2005\/gmd-noaa<\/li>\n<li>eml:\/\/ecoinformatics.org\/eml-2.1.1<\/li>\n<li>eml:\/\/ecoinformatics.org\/eml-2.1.0<\/li>\n<li>eml:\/\/ecoinformatics.org\/eml-2.0.1<\/li>\n<li>eml:\/\/ecoinformatics.org\/eml-2.0.0<\/li>\n<li>http:\/\/www.openarchives.org\/ore\/terms<\/li>\n<\/ol>\n<ul>\n<li><strong>Develop a unittest in python for testing the custom magic file.<\/strong><\/li>\n<\/ul>\n<p>A python unittest was created using the unittest library for functional testing of the magic file. This test helped us in making sure that when we are adding new patterns for other files it doesn&#8217;t break.<\/p>\n<ul>\n<li><strong>Compile the magic file with new changes, so that it can be distributed and used by others.<\/strong><\/li>\n<\/ul>\n<p>After testing and creation of the magic file we wanted to create and share the file with others. This can be done by passing the magic using -m option or can be compiled on linux using the <a href=\"https:\/\/github.com\/DataONEorg\/file_identification#installing-and-using-dataone-magic-file\">steps<\/a> mentioned in readme.<\/p>\n<ul>\n<li><strong>Submit the dataone magic file in Magic(5) format for inclusion with the libmagic library.<\/strong><\/li>\n<\/ul>\n<p>We submitted the dataone magic file in Magic(5) format to the Linux administrators for including the changes in the next release of file and libmagic. The changes were accepted and committed to the repository and will be available in the next release of the file command (probably file 5.34 and above.<\/p>\n<ul>\n<li><strong>Apace-Tika, and customization of the custom-mimetypes.xml file.<\/strong><\/li>\n<\/ul>\n<p>Apache-Tika is another tool, which can be used for&nbsp;detecting and extracting&nbsp;metadata and text from over a thousand different file types. It uses an xml file which contains all the information about the existing file-types. &nbsp;Our next step is to develop and pass the custom mime type information for the DataONE file formats, which can be detected using Tika.<\/p>\n<p>That&#8217;s all for now, see you all next week!<\/p>\n<p>Have a great weekend!<\/p>\n<p><span style=\"font-size: 15px\">Resource&nbsp;links: <\/span><a style=\"font-size: 15px\" href=\"https:\/\/github.com\/DataONEorg\/file_identification\">Github<\/a><span style=\"font-size: 15px\">,<\/span><a style=\"font-size: 15px\" href=\"https:\/\/hpad.dataone.org\/2018-internship-fileidentity?both\">Project&nbsp;Plan<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi All, This is the second week of my internship, and&nbsp;below are the tasks that were completed during this week. Adding the patterns for the rest of the file formats into the dataone magic file. Continuing the work from the last week, we were able to create additional patterns for <a class=\"more-link\" href=\"https:\/\/notebooks.dataone.org\/extending-libmagic\/week-2-created-tested-and-committed-magic-file-with-the-libmagic-library\/\">Continue reading <span class=\"screen-reader-text\">  Week 2: Created, tested and Committed magic file with the Libmagic library.<\/span><span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":113,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[389],"tags":[],"_links":{"self":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts\/3183"}],"collection":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/users\/113"}],"replies":[{"embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/comments?post=3183"}],"version-history":[{"count":4,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts\/3183\/revisions"}],"predecessor-version":[{"id":3351,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/posts\/3183\/revisions\/3351"}],"wp:attachment":[{"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/media?parent=3183"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/categories?post=3183"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/notebooks.dataone.org\/wp-json\/wp\/v2\/tags?post=3183"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}