Updating elasticsearch to a newer version has been quite a breeze in the past. But with the arrival of the new major es-release 2.0, I thought a bit more thorough test were in order. A good read for breaking changes can be found here. A very useful es-plugin to check your existing indexes for compatibility issues can be found here.
One of the major changes that did hit us, is the fact that field names may no longer contain dots. We use the elk stack to mainly log webserver logs. For some log types we also parse the URL get params to param_name => param_value pairs. Sadly, some of those param_names contain a dot, e.g. „document.x=123&document.y=345“. So, to migrate the exiting indexes we need to get rid of the dots in those fields.
Here is the LumberMill configuration that will reindex and replace dots with underscores:
# Sets number of parallel LumberMill processes.
# Recursively replace dots with underscores in all fieldnames below field "params".
# Copy old event type to new event.
# Drop internal es fields prior to reindex.
source_fields: [‚_uid‘, ‚_id‘, ‚_type‘, ‚_source‘, ‚_all‘, ‚_parent‘, ‚_field_names‘, ‚_routing‘, ‚_index‘, ‚_size‘, ‚_timestamp‘, ‚_ttl‘, ‚_score‘]