Import json data to hosted elasticsearch

Posted on in elk, tools, elastic

Earlier I imported the data to locally running Elastic (via Docker)

But others in my team could not run Docker on their machines, so we decided to use a hosted Elastic server.

The steps to import the data were the same. Well, almost. Because this was a hosted instance, it would not allow operations without authentication. I got the following error:

dump ended with error (get phase) => UNAUTHORIZED: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

It took me a while to figure it out, but I did.

To import the data, I had to pass the ApiKey as a header to elasticdump as follows:

node_modules/elasticdump/bin/elasticdump --input=formatted-conversations.json --output=https://some-real-hostname.regionosting-provider.elastic-cloud.com/ --headers='{"Authorization": "ApiKey here_is_my_Api_key_that_I_got_from_Kibana"}'

Creating an API Key was easy in Kibana. I searched for API Key in the top search bar 😄. (Management -> Security -> API Keys)

References:

  1. Elasticsearch Authentication using API
  2. Elasticdump README : Search for --headers