Updating elasticsearch to a newer version has been quite a breeze in the past. But with the arrival of the new major es-release 2.0, I thought a bit more thorough test were in order. A good read for breaking changes can be found here. A very useful es-plugin to check your existing indexes for compatibility issues can be found here.
One of the major changes that did hit us, is the fact that field names may no longer contain dots. We use the elk stack to mainly log webserver logs. For some log types we also parse the URL get params to param_name => param_value pairs. Sadly, some of those param_names contain a dot, e.g. „document.x=123&document.y=345“. So, to migrate the exiting indexes we need to get rid of the dots in those fields.
Here is the LumberMill configuration that will reindex and replace dots with underscores:
# Sets number of parallel LumberMill processes. - Global: workers: 3 - ElasticSearch: nodes: ['ELASTICSEARCH_HOST:9200'] search_type: scan index_name: 'INDEX_NAME' # Recursively replace dots with underscores in all fieldnames below field "params". - ModifyFields: action: rename_replace source_field: params recursive: True old: . new: _ # Copy old event type to new event. - ModifyFields: action: insert target_field: lumbermill.event_type value: $(_type) # Drop internal es fields prior to reindex. - ModifyFields: action: delete source_fields: ['_uid', '_id', '_type', '_source', '_all', '_parent', '_field_names', '_routing', '_index', '_size', '_timestamp', '_ttl', '_score'] - SimpleStats - ElasticSearchSink: nodes: ['ELASTICSEARCH_HOST:9200'] index_name: 'NEW_INDEX_NAME'