XML files in LogStash

XML files in LogStash

logstash-logoHow to import my custom XML files in ElasticSearch, luckily Logstash is there to help. Let’s create an example XML file that we want to import in Elasticsearch. Copy the text below and save this as ‘.xml’, you can also use your own XML.

<xmldata>
 <head1>
  <key1>Value1</key1>
  <key2>Value2</key2>
  <id>0001</id>
  <date>01-01-2016 09:00:00</date>
 </head1>
 <head2>
  <key3>Value3</key3>
 </head2>
</xmldata>

Building the Logstash config file

XML files can be readed by Logstash with the multi line option under the input file. Below I show my example configuration file;

input {
 file {
  path => "/config-dir/test.xml"
  start_position => beginning
  codec => multiline
  {
   pattern => "^<\?xmldata .*\>"
   negate => true
   what => "previous"
  }
 }
}

The filter part of the configuration will read the XML. This example will filter out specific values; Id, Date and Key1. Also the date value will be translated to correctly be used by Elasticsearch and Kibana.

filter {
  xml {
   store_xml => false
   source => "message"
   xpath =>
   [
    "/xmldata/head1/id/text()", "id",
    "/xmldata/head1/date/text()", "date",
    "/xmldata/head1/key1/text()", "key1"
   ]
}

date {
    match => [ "date" , "dd-MM-yyyy HH:mm:ss" ]
    timezone => "Europe/Amsterdam"
}

}

Under the output part we configure Elasticsearch and the stout so we can see the output directly in the console. In this example the Elasticsearch document_id is used from the XML file. Optional you can set the document_type part.

 
output { 
 stdout { codec => rubydebug } 
 elasticsearch { 
  index => "logstash-xml"
  hosts => ["10.0.1.3:9200"] 
  document_id => "%{[id]}"
  document_type => "xmlfiles" 
 } 
} 

Running LogStash

When you put the three pieces together, input/filter/output, you have the complete configuration file for LogStash. Save this file as logstash-xml.conf, you can test the config with the following command:

bin/logstash -f logstash-xml.conf --configtest

And run the config file with;

bin/logstash -f logstash-xml.conf

3 thoughts on “XML files in LogStash

  1. Hi Carlos,
    I thought this article was well written, especially for beginners like myself.
    I have a question about xpath.
    Say you have multiple tags with the same attribute name, with different attribute values. How would you address those?

    Value1
    Value2
    0001
    01-01-2016 09:00:00

    Value3

      1. Thanks for you comment on my blogpost, good to hear it’s readable. Let me know if you have further questions.

        Regards,
        Carlos

Comments are closed.

Comments are closed.