Designing Resilient Microservices with C# and ASP.NET Core

When you hear the word microservice, what comes to your mind? Before I learnt about microservices, I though they were cute and tiny looking services that made up a huge service/application. While I still think they are tiny and cute, there’s clearly more to them.

Microservices are a way to build software applications by breaking them into smaller, independent parts, called "services." Each service handles a specific feature or function of the application, and they all work together to create the full system.

Pre-requisites

To fully benefit from this article, readers should have the following prerequisites:

• Basic Knowledge of C# and ASP.NET Core

  • Understanding of C# syntax and object-oriented programming.

  • Familiarity with creating and running basic applications using ASP.NET Core.

• Fundamentals of Web APIs

  • Experience building and consuming RESTful APIs.

  • Understanding HTTP methods (GET, POST, PUT, DELETE) and status codes.

• Understanding of Microservices Architecture

  • Basic knowledge of microservices and how they differ from monolithic applications.

  • Familiarity with concepts like service boundaries, inter-service communication, and scaling.

• Experience with Dependency Injection and Middleware

  • Understanding how dependency injection works in ASP.NET Core.

  • Familiarity with using and configuring middleware in the ASP.NET Core pipeline.

• Experience with Docker and Containers (Optional but Beneficial)

  • Understanding containerization concepts and how to run applications in Docker.

  • Familiarity with setting up and deploying containerized applications.

• Basic Knowledge of Asynchronous Programming in C#

  • Understanding async/await patterns for non-blocking operations.

• Introduction to Cloud Services and Hosting (Optional)

  • Familiarity with cloud services like Azure or AWS, especially for hosting microservices.

Table of Contents

• Introduction to Microservices and Resilience

• Patterns for Microservice Resilience

• Implementing Circuit Breakers, Retries, and Fallbacks

• Case Studies and Best Practices

• Getting Started: Tools and Frameworks

• Conclusion and Additional Resources

Introduction to Microservices and Resilience

What Are Microservices?

Microservices are a way to build software applications by breaking them into smaller, independent parts, called "services." Each service handles a specific feature or function of the application, and they all work together to create the full system.

Key Characteristics of Microservices:

1. Independence: Each microservice runs separately and can be developed, deployed, and scaled independently.

2. Single Responsibility: Each service focuses on doing one thing well, like handling user accounts, payments, or product catalogs.

3. Communication via APIs: Microservices communicate with each other through well-defined APIs (often REST APIs over HTTP).

Example: Imagine an online store where different microservices manage:

• Products

• Orders

• Users

• Payments

Each of these services works independently but communicates with each other to form the complete system.

// Example of a simple Product microservice API controller in ASP.NET Core
[ApiController]
[Route("api/[controller]")]
public class ProductController : ControllerBase
{
    [HttpGet]
    public IActionResult GetProducts()
    {
        // Return a list of products
        return Ok(new List<string> { "Laptop", "Smartphone", "Tablet" });
    }

    [HttpPost]
    public IActionResult AddProduct(string product)
    {
        // Add a new product (logic omitted for simplicity)
        return Ok($"Product {product} added successfully.");
    }
}

Why Is Resilience Important in Microservices Architecture?

In microservices architecture, many services depend on each other. If one service fails, it can affect other parts of the system, making it harder to provide a good experience to users.

Resilience means that even if something goes wrong, the system continues to function as smoothly as possible. Resilience helps microservices:

1. Handle Failures Gracefully: If one service is down, others should keep working and return fallback responses or retry failed operations.

2. Avoid Cascading Failures: A failure in one service shouldn’t bring down the entire application.

3. Ensure Reliability: Even when things go wrong (like network issues), the system still provides a good user experience.

Example of Resilience: Imagine your online store's Payment service goes down. Without resilience, users won't be able to check out, and the whole system might stop functioning. With resilience techniques like retries or fallbacks, the system could inform users of a delay or try again later.

Basic Example of Resilience: Retry Policy

One way to handle failures is to retry an operation (like calling another service) a few times before giving up.

// Example of implementing a simple retry policy using Polly (a .NET resilience library)
public async Task<string> CallPaymentServiceAsync()
{
    var retryPolicy = Policy
        .Handle<HttpRequestException>()
        .RetryAsync(3); // Retry up to 3 times if an exception occurs

    return await retryPolicy.ExecuteAsync(async () =>
    {
        // Call the Payment service
        var response = await httpClient.GetStringAsync("https://payment-service/api/payment");
        return response;
    });
}

In this example, if the Payment service fails (throws an exception), the system will automatically retry up to 3 times before returning an error. This makes the system more resilient to temporary failures.

Conclusion: Microservices provide flexibility and scalability, but they also introduce complexity and the potential for failures. By building resilience into your microservices architecture, you ensure that your system continues to function smoothly, even when some parts fail.

Patterns for Microservice Resilience

In our 17th article in the Mastering C series, we have an indepth guide discussing how to implement resilience patterns. When building microservices, resilience is essential to ensure that your system can handle failures gracefully and keep working. Let’s look at some common resilience patterns and how to apply them in C# and ASP.NET Core.

Retry Pattern

The Retry Pattern automatically retries a failed operation (like a network request) a set number of times before giving up. This helps handle temporary failures, such as network issues or service downtime.

Example: Retry with Polly Library

You can use the Polly library in C# to easily implement retries. Install it using NuGet:

Install-Package Polly

Now, let’s apply a retry policy:

using Polly;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class MicroserviceClient
{
    private static HttpClient _client = new HttpClient();

    public async Task<string> GetDataWithRetryAsync(string url)
    {
        // Define a retry policy with 3 retries
        var retryPolicy = Policy
            .Handle<HttpRequestException>()
            .RetryAsync(3, onRetry: (exception, retryCount) =>
            {
                Console.WriteLine($"Retry {retryCount} for {url}");
            });

        // Execute the retry policy
        return await retryPolicy.ExecuteAsync(async () =>
        {
            var response = await _client.GetAsync(url);
            response.EnsureSuccessStatusCode(); // Throw if not successful
            return await response.Content.ReadAsStringAsync();
        });
    }
}

In this code:

• If the request fails (e.g., due to a network issue), it will retry up to 3 times.

• The RetryAsync method retries only when an exception (like HttpRequestException) is caught.

Circuit Breaker Pattern

The Circuit Breaker Pattern prevents repeated calls to a failing service by "breaking" the connection after several failures. After a while, it attempts to restore the connection.

Example: Circuit Breaker with Polly

Here’s how to use Polly for a circuit breaker:

using Polly;
using System.Net.Http;
using System.Threading.Tasks;

public class MicroserviceClient
{
    private static HttpClient _client = new HttpClient();

    // Circuit breaker breaks after 2 failures and resets after 30 seconds
    private static IAsyncPolicy _circuitBreakerPolicy = Policy
        .Handle<HttpRequestException>()
        .CircuitBreakerAsync(2, TimeSpan.FromSeconds(30));

    public async Task<string> GetDataWithCircuitBreakerAsync(string url)
    {
        try
        {
            return await _circuitBreakerPolicy.ExecuteAsync(async () =>
            {
                var response = await _client.GetAsync(url);
                response.EnsureSuccessStatusCode(); // Throw if not successful
                return await response.Content.ReadAsStringAsync();
            });
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Circuit is open: {ex.Message}");
            throw;
        }
    }
}

• After 2 failures, the circuit breaker opens and will stop trying for 30 seconds.

• If you try again during that time, the operation will fail immediately, preventing overloading of the failing service.

Fallback Pattern

The Fallback Pattern provides a default response when a service fails, ensuring the system still returns something useful.

Example: Fallback with Polly

You can define a fallback response like this:

using Polly;
using System.Net.Http;
using System.Threading.Tasks;

public class MicroserviceClient
{
    private static HttpClient _client = new HttpClient();

    public async Task<string> GetDataWithFallbackAsync(string url)
    {
        // Fallback to a default message if the service call fails
        var fallbackPolicy = Policy<string>
            .Handle<HttpRequestException>()
            .FallbackAsync("Default fallback response");

        return await fallbackPolicy.ExecuteAsync(async () =>
        {
            var response = await _client.GetAsync(url);
            response.EnsureSuccessStatusCode(); // Throw if not successful
            return await response.Content.ReadAsStringAsync();
        });
    }
}

• If the request fails, the fallback policy returns "Default fallback response" instead of crashing or returning nothing.

Timeout Pattern

The Timeout Pattern ensures that an operation doesn't hang indefinitely by setting a maximum time limit.

Example: Timeout with Polly

Here's how to implement a timeout:

using Polly;
using System.Net.Http;
using System.Threading.Tasks;

public class MicroserviceClient
{
    private static HttpClient _client = new HttpClient();

    public async Task<string> GetDataWithTimeoutAsync(string url)
    {
        // Define a 5-second timeout policy
        var timeoutPolicy = Policy.TimeoutAsync(5); // 5 seconds

        return await timeoutPolicy.ExecuteAsync(async () =>
        {
            var response = await _client.GetAsync(url);
            response.EnsureSuccessStatusCode(); // Throw if not successful
            return await response.Content.ReadAsStringAsync();
        });
    }
}

• If the request takes longer than 5 seconds, it will be canceled, preventing potential hangs.

Combining Patterns

You can combine these patterns to build even more resilient microservices. For example, you could combine Retry with Circuit Breaker and Timeout to handle various failure scenarios effectively.

Example: Combining Retry, Circuit Breaker, and Timeout

public async Task<string> GetDataWithCombinedPoliciesAsync(string url)
{
    // Retry 3 times, break after 2 failures, and apply a 5-second timeout
    var combinedPolicy = Policy.WrapAsync(
        Policy.TimeoutAsync(5), // 5-second timeout
        Policy.Handle<HttpRequestException>().RetryAsync(3), // Retry 3 times
        Policy.Handle<HttpRequestException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(30)) // Circuit breaker
    );

    return await combinedPolicy.ExecuteAsync(async () =>
    {
        var response = await _client.GetAsync(url);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    });
}

In this example:

• The request has a timeout of 5 seconds.

• If a request fails, it will retry up to 3 times.

• If there are 2 consecutive failures, the circuit breaker will open for 30 seconds.

Conclusion: By applying these patterns (Retry, Circuit Breaker, Fallback, and Timeout), you can make your microservices more resilient to failures, ensuring they can handle temporary issues and continue functioning smoothly. Libraries like Polly make it easy to implement these patterns in C# and ASP.NET Core.

Implementing Circuit Breakers, Retries, and Fallbacks

When building microservices, we want to ensure they can handle failures gracefully. Circuit breakers, retries, and fallbacks are key patterns for making microservices more resilient. Let’s break these down in a simple way.

What is a Circuit Breaker, and When Should You Use It?

A circuit breaker is like a safety switch. When your microservice tries to call another service (e.g., a payment service) and that service is slow or failing, the circuit breaker steps in. Instead of repeatedly trying and wasting time, it "breaks" the connection after too many failures. It prevents overwhelming the failing service and helps your system recover faster.

When to Use a Circuit Breaker:

• When calling an external service that might fail.

• To avoid sending requests to a service that is already failing.

• To allow your service to recover more quickly from failure.

Example: Implementing a Circuit Breaker

Using Polly, a popular .NET library, you can implement a circuit breaker in a few lines of code. Here’s an example of how to set it up in your ASP.NET Core microservice.

using Polly;
using Polly.CircuitBreaker;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class CircuitBreakerExample
{
    private static AsyncCircuitBreakerPolicy<HttpResponseMessage> _circuitBreakerPolicy = 
        Policy.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
        .CircuitBreakerAsync(2, TimeSpan.FromSeconds(30));

    public async Task<HttpResponseMessage> CallExternalServiceAsync(HttpClient httpClient, string url)
    {
        return await _circuitBreakerPolicy.ExecuteAsync(() => httpClient.GetAsync(url));
    }
}

In this code:

• The circuit breaker "trips" after 2 failed attempts.

• Once tripped, it stays open for 30 seconds, during which any request will be short-circuited, meaning it won’t even try the failing service.

How to Implement Retries in a Microservice

Retries are useful when a failure might be temporary. Instead of failing immediately, you can try the operation a few more times before giving up. This is helpful when you think the issue may resolve itself (e.g., a network glitch).

Example: Implementing Retries

Here’s how to add retries using Polly. In this example, we try the external service call up to 3 times, waiting 2 seconds between each attempt.

using Polly;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class RetryExample
{
    private static AsyncRetryPolicy<HttpResponseMessage> _retryPolicy = 
        Policy.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(2));

    public async Task<HttpResponseMessage> CallExternalServiceWithRetryAsync(HttpClient httpClient, string url)
    {
        return await _retryPolicy.ExecuteAsync(() => httpClient.GetAsync(url));
    }
}

In this code:

• We retry up to 3 times.

• After each failed attempt, we wait 2 seconds before trying again.

Using Fallbacks to Ensure System Reliability

A fallback is what your system does when everything else fails. For example, if an external service isn’t working, instead of crashing, your system can return a default response or run some alternative logic to keep things running smoothly.

Example: Implementing Fallbacks

Here’s how you can use Polly to define a fallback. If all retries or circuit breakers fail, the fallback kicks in with a predefined response.

using Polly;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class FallbackExample
{
    private static AsyncFallbackPolicy<HttpResponseMessage> _fallbackPolicy =
        Policy<HttpResponseMessage>
        .Handle<HttpRequestException>()
        .OrResult(r => !r.IsSuccessStatusCode)
        .FallbackAsync(new HttpResponseMessage(System.Net.HttpStatusCode.OK) 
        { 
            Content = new StringContent("Fallback response due to failure.") 
        });

    public async Task<HttpResponseMessage> CallExternalServiceWithFallbackAsync(HttpClient httpClient, string url)
    {
        return await _fallbackPolicy.ExecuteAsync(() => httpClient.GetAsync(url));
    }
}

In this code:

• If the service call fails (due to a network error or a non-successful status code), the fallback provides a safe, default response ("Fallback response due to failure.").

• This ensures your system keeps working even when external services are down.

When to Use These Patterns Together

• Circuit breakers protect your system from wasting resources on failing services.

• Retries help deal with temporary issues by trying again.

• Fallbacks provide a safety net to keep your system running smoothly even when something goes wrong.

By using these patterns together, you can make your microservices much more reliable and resilient, ensuring that your system can handle failures without crashing or slowing down.

Case Studies and Best Practices

When designing resilient microservices, it’s essential to understand real-world challenges and solutions. Here are some simple case studies and best practices to guide you in creating fault-tolerant microservices with C# and ASP.NET Core.

Case Study 1: Circuit Breaker Pattern

Problem: A payment service in an e-commerce system frequently fails due to a third-party payment gateway going down. This causes delays and timeouts, making the entire system unreliable.

Solution: Implement a circuit breaker pattern. This pattern "breaks" the connection when a service consistently fails and retries after a specified time.

Code Example: Circuit Breaker Using Polly

using Polly;
using Polly.CircuitBreaker;
using System.Net.Http;

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreaker(2, TimeSpan.FromMinutes(1)); // Break after 2 failures, retry after 1 minute

var httpClient = new HttpClient();

try
{
    await circuitBreakerPolicy.ExecuteAsync(async () => 
    {
        var response = await httpClient.GetAsync("https://api.paymentgateway.com/charge");
        response.EnsureSuccessStatusCode();
    });
}
catch (BrokenCircuitException)
{
    Console.WriteLine("Circuit is broken, please try again later.");
}

Best Practice:

• Use circuit breakers to protect your microservices from cascading failures due to unreliable external services.

Case Study 2: Retry Pattern

Problem: A shipping service is occasionally unavailable due to high traffic but recovers quickly. However, failures lead to incomplete orders in the system.

Solution: Use the retry pattern to attempt requests a few times before giving up, allowing for temporary service outages.

Code Example: Retry Using Polly

using Polly;
using System.Net.Http;

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .RetryAsync(3); // Retry 3 times before failing

var httpClient = new HttpClient();

await retryPolicy.ExecuteAsync(async () => 
{
    var response = await httpClient.GetAsync("https://api.shipping.com/process");
    response.EnsureSuccessStatusCode();
});

Best Practice:

• Use retries for transient failures, ensuring your microservices can recover from temporary issues without user intervention.

Case Study 3: Fallback Pattern

Problem: A product service that fetches item details from an inventory service occasionally fails due to a network issue. When this happens, the user is left with incomplete product information.

Solution: Implement a fallback to return default data or a cached response when the primary service is unavailable.

Code Example: Fallback Using Polly

using Polly;
using System.Net.Http;

var fallbackPolicy = Policy<string>
    .Handle<HttpRequestException>()
    .FallbackAsync("Product details currently unavailable."); // Fallback message

var httpClient = new HttpClient();

var result = await fallbackPolicy.ExecuteAsync(async () => 
{
    var response = await httpClient.GetStringAsync("https://api.inventory.com/product/123");
    return response;
});

Console.WriteLine(result);

Best Practice:

• Use fallbacks to ensure your microservices provide a reasonable response, even when services fail, enhancing the user experience.

Best Practices for Designing Fault-Tolerant Microservices

• Decoupling Services:

  • Ensure that services can function independently as much as possible. Avoid tight coupling between microservices, so failures don’t cascade through the system.

• Timeouts and Retries:

  • Always set timeouts when making HTTP calls between microservices to avoid long waits in case of failure.

  • Combine retries with exponential backoff to reduce load on failing services.

Example:

    var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromSeconds(5)); // 5-second timeout

• Graceful Degradation:

  • Have a backup plan (like a cached response) to maintain functionality when some services are down. This keeps users engaged without seeing broken pages or missing data.

• Monitoring and Alerts:

  • Implement monitoring to track service health and set up alerts for failures. Tools like Application Insights or Prometheus can help.

• Use Asynchronous Communication:

  • Prefer asynchronous messaging (using queues like RabbitMQ or Kafka) for communication between microservices to improve resilience and avoid bottlenecks.

Getting Started: Tools and Frameworks

To begin building resilient microservices with C# and ASP.NET Core, you'll need a few key tools and libraries to ensure your services can handle failures gracefully. Here's a simple overview:

Tools and Libraries for Building Resilient Microservices

• ASP.NET Core:
  This is the main framework you'll use to build your microservices. It provides a powerful, flexible environment to create web APIs that are the backbone of microservices.

• Polly:
  A popular .NET library that helps you implement resilience patterns like retries, circuit breakers, and fallbacks. It allows your services to recover from transient failures.

• Docker:
  Docker helps you package your microservices into containers, making them easy to deploy and manage consistently across different environments.

• Swagger:
  A tool for generating documentation and testing your APIs. It makes it easy to explore and interact with your microservices.

• Serilog:
  A logging library to help you track errors and monitor the health of your microservices.

Setting Up Your First Resilient Microservice in ASP.NET Core

1. Install the Tools:

   • Install Visual Studio or Visual Studio Code as your development environment.

   • Set up .NET SDK and ASP.NET Core.

2. Create a New ASP.NET Core Web API:

   • Open your terminal or IDE and run:

       dotnet new webapi -n ResilientMicroservice
     

   • This command will create a new web API project named "ResilientMicroservice."

3. Add Polly for Resilience:

   • Add the Polly library using NuGet:

       dotnet add package Polly
     

   • You can now start adding retries, circuit breakers, and other resilience strategies to your microservice.

4. Test Your Service with Swagger:

   • ASP.NET Core automatically adds Swagger when you create a new API project. Run the project with:

       dotnet run
     

   • Open the Swagger interface in your browser and test your endpoints to see how they work.

These tools and steps will help you build a resilient microservice that can handle potential issues while remaining stable and reliable.

Conclusion and Additional Resources

In this guide, we’ve explored how to design resilient microservices using C# and ASP.NET Core. We covered key patterns like circuit breakers, retries, and fallbacks that help your microservices handle failures gracefully. By applying these patterns, you can build systems that stay reliable, even when things go wrong.

Additional Resources

To learn more and deepen your understanding, here are some helpful resources:

• Official ASP.NET Core Documentation: A great place to learn more about ASP.NET Core and how to use it effectively.

• Microservices with ASP.NET Core: Microsoft's guide on building microservices.

• Designing Distributed Systems by Brendan Burns: A helpful book on microservices and distributed systems.

• Polly Documentation: The library we used for implementing resilience patterns like retries and circuit breakers.

Keep learning and experimenting with these patterns in your projects, and you'll continue to improve your microservices. I hope you found this guide helpful and learned something new. Stay tuned for the next article in the Mastering C# series: Exploring Roslyn: The .NET Compiler Platform

Happy coding!