Introduction  In today's data-driven world, having efficient search capabilities within your application is crucial. Elasticsearch, an open-source, distributed search and analytics engine, is designed for this purpose. Coupled with Kibana, a powerful visualization tool, you can not only search through large datasets quickly but also visualize and analyze your data in real-time. This blog post will guide you through integrating Elasticsearch and Kibana into your .NET Core application, focusing on setting up efficient search capabilities.   What is Elasticsearch?  Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Key features include: Distributed: Elasticsearch distributes data and processing across multiple nodes, ensuring high availability and scalability. Full-text search: It offers powerful full-text search capabilities, including complex search queries. Real-time indexing and searching: Elasticsearch provides near real-time search capabilities, making it ideal for applications that require up-to-date search results. RESTful API: Elasticsearch's API is RESTful, making it easy to interact with using HTTP requests.   What is Kibana?  Kibana is an open-source data visualization and exploration tool designed to work with Elasticsearch. It provides a web interface for: Visualizing Elasticsearch data: Create charts, graphs, and maps to visualize your data. Exploring data: Use the Discover feature to explore your indexed data and perform searches. Creating dashboards: Combine multiple visualizations into interactive dashboards for monitoring and analysis. Real-time monitoring: Monitor your data and set up alerts for specific events or conditions.   Prerequisites  Before we start, ensure you have the following: Elasticsearch: Installed and running. Download from the official Elasticsearch website. Kibana: Installed and running. Download from the official Kibana website. .NET Core 6 or higher: Installed and ready for development. Lower versions of .NET Core are supported, but .NET Core 6 is the latest LTS version, supported until November 2024. .NET Core 8 is expected to be the next LTS release in November 2023, with support through November 2026.   Setting Up Elasticsearch in .NET Core  Step 1: Create a New .NET Core Project WebApp or API Step 2: Add Elasticsearch NuGet Packages dotnet add package Elasticsearch.Net dotnet add package NEST Step 3: Configure Elasticsearch Add Configuration: In appsettings.json, add your Elasticsearch URL: {   "Elasticsearch": {     "Url": "http://localhost:9200"   } } Create Elasticsearch Service: Create a service to handle Elasticsearch interactions. using Elasticsearch.Net; using Microsoft.Extensions.Configuration; using Nest; using System; public class ElasticsearchService {     private readonly IElasticClient _elasticClient;     public ElasticsearchService(IConfiguration configuration)     {         var settings = new ConnectionSettings(new Uri(configuration["Elasticsearch:Url"]))                        .DefaultIndex("default-index");         _elasticClient = new ElasticClient(settings);     }     public async Task IndexDocumentAsync<T>(T document) where T : class     {         await _elasticClient.IndexDocumentAsync(document);     }     public async Task<ISearchResponse<T>> SearchAsync<T>(Func<SearchDescriptor<T>, ISearchRequest> searchSelector) where T : class     {         return await _elasticClient.SearchAsync(searchSelector);     } } Register Elasticsearch Service: Register the service in Startup.cs or Program.cs. public class Startup {     public void ConfigureServices(IServiceCollection services)     {         services.AddControllers();         services.AddSingleton<ElasticsearchService>();     }     public void Configure(IApplicationBuilder app, IWebHostEnvironment env)     {         if (env.IsDevelopment())         {             app.UseDeveloperExceptionPage();         }         app.UseRouting();         app.UseEndpoints(endpoints =>         {             endpoints.MapControllers();         });     } } Use Elasticsearch in Controllers: Create a controller to interact with Elasticsearch. using Microsoft.AspNetCore.Mvc; using System.Threading.Tasks; [ApiController] [Route("[controller]")] public class SearchController : ControllerBase {     private readonly ElasticsearchService _elasticsearchService;     public SearchController(ElasticsearchService elasticsearchService)     {         _elasticsearchService = elasticsearchService;     }     [HttpPost("index")]     public async Task<IActionResult> IndexDocument([FromBody] object document)     {         await _elasticsearchService.IndexDocumentAsync(document);         return Ok();     }     [HttpGet("search")]     public async Task<IActionResult> Search(string query)     {         var response = await _elasticsearchService.SearchAsync<object>(s => s             .Query(q => q                 .QueryString(d => d                     .Query(query)                 )             )         );         return Ok(response.Documents);     } } Step 4: Index Existing Data If you have existing data in your database, you'll need to index it into Elasticsearch. Create a Data Indexing Service: using Microsoft.Extensions.Configuration; using Nest; using System; using System.Collections.Generic; using System.Threading.Tasks; public class DataIndexingService {     private readonly IElasticClient _elasticClient;     public DataIndexingService(IConfiguration configuration)     {         var settings = new ConnectionSettings(new Uri(configuration["Elasticsearch:Url"]))                        .DefaultIndex("your-index-name");         _elasticClient = new ElasticClient(settings);     }     public async Task IndexDataAsync<T>(IEnumerable<T> data) where T : class     {         var bulkDescriptor = new BulkDescriptor();         foreach (var item in data)         {             bulkDescriptor.Index<T>(op => op                 .Document(item)             );         }         var response = await _elasticClient.BulkAsync(bulkDescriptor);         if (response.Errors)         {             throw new Exception("Failed to index some documents");         }     } } Load Existing Data: Fetch data from your database. using System.Collections.Generic; using System.Threading.Tasks; using Dapper; using Microsoft.Data.SqlClient; public class DatabaseService {     private readonly string _connectionString;     public DatabaseService(string connectionString)     {         _connectionString = connectionString;     }     public async Task<IEnumerable<YourDataType>> GetExistingDataAsync()     {         using (var connection = new SqlConnection(_connectionString))         {             var query = "SELECT * FROM YourTable";             var data = await connection.QueryAsync<YourDataType>(query);             return data;         }     } } Index Existing Data: In Startup.cs or Program.cs, index your existing data at startup. public class Startup {     public void ConfigureServices(IServiceCollection services)     {         services.AddSingleton<DatabaseService>(sp => new DatabaseService("YourConnectionString"));         services.AddSingleton<DataIndexingService>();         services.AddControllers();     }     public void Configure(IApplicationBuilder app, IWebHostEnvironment env, DatabaseService databaseService, DataIndexingService dataIndexingService)     {         // Index existing data at startup         Task.Run(async () =>         {             var existingData = await databaseService.GetExistingDataAsync();             await dataIndexingService.IndexDataAsync(existingData);         }).GetAwaiter().GetResult();         if (env.IsDevelopment())         {             app.UseDeveloperExceptionPage();         }         app.UseRouting();         app.UseEndpoints(endpoints =>         {             endpoints.MapControllers();         });     } } Step 1: Navigate to kibana.yml and Update Configurations Open Command Prompt (cmd): Press Win + R, type cmd, and press Enter. Navigate to the Kibana Directory: Use the cd command to navigate to the directory where Kibana is installed. For example: cd C:\path\to\kibana\config Edit kibana.yml: Open kibana.yml in a text editor. You can use notepad from the command line: notepad kibana.yml Update the Configuration: In the kibana.yml file, set the elasticsearch.hosts property to point to your Elasticsearch instance: elasticsearch.hosts: ["http://localhost:9200"] Save and close kibana.yml. Step 2: Run elasticsearch.bat and kibana.bat Using Command Prompt Open a New Command Prompt (cmd) Window for Elasticsearch: Press Win + R, type cmd, and press Enter. Navigate to the Elasticsearch bin Directory: Use the cd command to navigate to the directory where Elasticsearch is installed. For example: cd C:\path\to\elasticsearch\bin Run elasticsearch.bat: Start Elasticsearch by running: elasticsearch.bat Open Another Command Prompt (cmd) Window for Kibana: Press Win + R, type cmd, and press Enter. Navigate to the Kibana bin Directory: Use the cd command to navigate to the directory where Kibana is installed. For example: cd C:\path\to\kibana\bin Run kibana.bat: Start Kibana by running: kibana.bat Step 3: Set the Password for Default User elastic Open a New Command Prompt (cmd) Window: Press Win + R, type cmd, and press Enter. Navigate to the Elasticsearch bin Directory: Use the cd command to navigate to the directory where Elasticsearch is installed. For example: cd C:\path\to\elasticsearch\bin Set the Password for the elastic User: Use the elasticsearch-users tool to set the password. Run the following command: elasticsearch-users userpasswd elastic You will be prompted to enter a new password for the elastic user.   Verification  Verify Elasticsearch and Kibana: Open your browser and navigate to http://localhost:9200 to check if Elasticsearch is running. Navigate to http://localhost:5601 to check if Kibana is running. Log in to Kibana: Use the elastic user and the password you set to log in to Kibana.   Exploring Data with Kibana  Kibana provides a web interface to visualize and explore your Elasticsearch data. After indexing your data, follow these steps: Step 1: Access Kibana Open your browser and navigate to http://localhost:5601. Step 2: Configure Index Pattern Go to Management > Kibana > Index Patterns. Create a new index pattern matching your indices, e.g., your-index-name-*. Step 3: Visualize Data Use the Discover tab to explore your indexed data and perform searches. Create visualizations using the Visualize tab: Choose a visualization type (e.g., bar chart, pie chart, line graph). Configure the data source and settings. Save the visualization. Step 4: Create Dashboards Combine multiple visualizations into interactive dashboards: Go to the Dashboard tab. Create a new dashboard. Add saved visualizations and arrange them as needed. Save the dashboard. Step 5: Real-Time Monitoring Set up real-time monitoring and alerts: Use the Monitoring feature to track the health and performance of your Elasticsearch cluster. Set up Watchers in Kibana to trigger alerts based on specific conditions.   Refer to the image below, which show the amount of data in the table     Conclusion  Integrating Elasticsearch and Kibana with your .NET Core application provides powerful search and visualization capabilities. With Elasticsearch, you can efficiently search through large datasets, and Kibana allows you to visualize and explore this data in real-time. By following the steps outlined in this blog post, you can enhance your application's search functionality and gain valuable insights from your data.