Introduction:
In the world of data analytics and log management, Elasticsearch and Kibana stand out as powerful tools for indexing, searching, and visualizing large volumes of data. In this tutorial, we’ll walk through the process of setting up Elasticsearch and Kibana on an EC2 instance, enabling you to harness the full potential of these tools for your projects.

Prerequisites:
Before we dive into the installation and setup process, ensure you have the following:

• An AWS account with access to EC2 services.
• Basic knowledge of working with Linux terminal commands.
• Access to an EC2 instance running a Linux distribution( ubuntu preferred ).

Installation Steps:

1. Update Package Repository: sudo apt-get update
2. Install Java Development Kit (JDK) 11: sudo apt-get install -y openjdk-11-jdk
3. Install Elasticsearch:
   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'
sudo apt-get update
sudo apt-get install -y elasticsearch
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
4. Install Kibana:
   sudo apt-get install -y kibana
sudo systemctl start kibana
sudo systemctl enable kibana
5. Configure Kibana:
   Open /etc/kibana/kibana.yml and uncomment or add the following lines:
   server.host: "0.0.0.0"
server.port: 5601
elasticsearch.username: "kibana_system"
elasticsearch.password: "password"
   
6. Restart Kibana Service:
   sudo systemctl restart kibana
7. Verify Connectivity:
   Ensure you can connect to Kibana by using Telnet:
   telnet <EC2_IP> 5601
8. Setup Passwords:
   Run the following command to initiate password setup for Elasticsearch users:
   cd /usr/share/elasticsearch sudo ./bin/elasticsearch-setup-passwords auto Follow the prompts to generate and set passwords for various users. The password will also be generated for kibana_system user. Replace this in the config file in step 5.
9. Enable Security in Elasticsearch:
   Add the following line to /etc/elasticsearch/elasticsearch.yml to enable security: xpack.security.enabled: true
   
   This step is required to add the login screen. Otherwise, the dashboard would load up without asking for any credentials.
   
10. Restart Elasticsearch Service:
    After making configuration changes, restart Elasticsearch and Kibana:
    sudo systemctl restart elasticsearch
sudo systemctl restart kibana
11. Additionally dont forget to expose the port 5601 via the inbound rules of security group in aws. Now you can visit IP:5601 and the kibana should load.

Logs

In case you run into any errors, you can view the Kibana logs at /var/log/kibana/kibana.log

Conclusion:
Congratulations! You’ve successfully set up Elasticsearch and Kibana on your EC2 instance. These powerful tools are now ready to help you manage, analyze, and visualize your data effectively. Explore further customization options and integrations to make the most out of your Elasticsearch and Kibana setup. Happy analyzing!