A Guide to Java Microservices. Part 3: general questions

Translation and adaptation of Java Microservices: A Practical Guide . Previous parts of the guide:

• microservices basics and their architecture ;
• deploying and testing microservices .

Let's look at the inherent problems of microservices in Java, starting with abstract things and ending with concrete libraries.

How to make a Java microservice resilient?

Recall that when you create microservices, you are essentially trading JVM method calls for synchronous HTTP calls or asynchronous messaging. While a method call is mostly guaranteed to complete (barring an unexpected JVM shutdown), a network call is unreliable by default. It may work, but it may not work for various reasons: the network is overloaded, a new firewall rule has been implemented, and so on. To see how this makes a difference, let's take a look at the BillingService example.

HTTP/REST Resilience Patterns

Let's say customers can buy e-books on your company's website. To do this, you've just implemented a billing microservice that can call your online store to generate actual PDF invoices. For now, we'll make this call synchronously, over HTTP (although it makes more sense to call this service asynchronously, since PDF generation doesn't have to be instantaneous from the user's perspective. We'll use the same example in the next section and look at the differences).

@Service
class BillingService {

    @Autowired
    private HttpClient client;

     public void bill(User user, Plan plan) {
        Invoice invoice = createInvoice(user, plan);
        httpClient.send(invoiceRequest(user.getEmail(), invoice), responseHandler());
        // ...
    }
}

To summarize, here are three possible outcomes of this HTTP call.

• OK: the call went through, the account was created successfully.
• DELAY: The call went through, but it took too long to complete.
• ERROR. The call failed, you may have sent an incompatible request, or the system may not be working.

Any program is expected to handle error situations, not just successful ones. The same applies to microservices. Even if you need to put in extra effort to ensure that all deployed versions of the API are compatible once you start with deployments and releases of individual microservices. An interesting case to look at is the case of delay. For example, the responder's microservice hard drive is full, and instead of taking 50 ms, it takes 10 seconds to respond. It gets even more interesting when you experience a certain load such that the unresponsiveness of your BillingService begins to cascade through your system. As an illustrative example, imagine a kitchen slowly starting up a “block” of all restaurant waiters. This section obviously cannot provide an exhaustive overview of the topic of microservice resilience, but it serves as a reminder to developers that this is actually something that needs to be addressed and not ignored before your first release (which, from experience, happens more often than not). follows). A popular library that helps you think about latency and fault tolerance is Netflix's Hystrix. Use its documentation to dive deeper into the topic.

Messaging Resilience Patterns

Let's take a closer look at asynchronous communication. Our BillingService program might now look something like this, assuming we're using Spring and RabbitMQ for messaging. To create an account, we now send a message to our RabbitMQ message broker, where there are several workers waiting for new messages. These workers create PDF invoices and send them to the appropriate users.

@Service
class BillingService {

    @Autowired
    private RabbitTemplate rabbitTemplate;

     public void bill(User user, Plan plan) {
        Invoice invoice = createInvoice(user, plan);
        // преобразует счет, например, в json и использует его How тело messages
        rabbitTemplate.convertAndSend(exchange, routingkey, invoice);
        // ...
    }
}

Potential errors look a little different now, as you no longer receive immediate OK or ERROR responses like you did with a synchronous HTTP connection. Instead, we may have three potential scenarios that could go wrong, which could raise the following questions:

1. Was my message delivered and used by the employee? Or is it lost? (The user does not receive an invoice).
2. Was my message only delivered once? Or delivered more than once and processed only once? (The user will receive multiple invoices).
3. Configuration: From "Did I use the correct routing keys/names for the exchange" to "Is my message broker configured and maintained correctly or are its queues full?" (The user does not receive an invoice).

A detailed description of each individual asynchronous microservice resilience pattern is beyond the scope of this guide. However, there are signs in the right direction. Moreover, they will depend on messaging technology. Examples:

• If you're using JMS implementations like ActiveMQ, you can trade speed for the guarantee of two-phase (XA) commits.
• If you're using RabbitMQ, read this tutorial first, and then think carefully about confirmations, fault tolerance, and message reliability in general.
• Perhaps someone is well versed in configuring Active or RabbitMQ servers, especially in combination with clustering and Docker (anyone? ;))

Which framework would be the best solution for Java microservices?

On the one hand, you can install a very popular option such as Spring Boot . It makes it very easy to create .jar files, comes with a built-in web server like Tomcat or Jetty, and can be run quickly and anywhere. Ideal for building microservice applications. Recently, a couple of specialized microservice frameworks, Kubernetes or GraalVM , appeared, partially inspired by reactive programming. Here are a few more interesting contenders: Quarkus , Micronaut , Vert.x , Helidon . In the end, you'll have to choose for yourself, but we can give you a couple of recommendations that may not be completely standard: With the exception of Spring Boot, all microservices frameworks are usually marketed as incredibly fast, with almost instantaneous startup, low memory usage, scalability to infinity. Marketing materials typically feature impressive graphics that show off the platform next to the behemoth Spring Boot or to each other. This, in theory, spares the nerves of developers supporting legacy projects, which sometimes take several minutes to load. Or developers working in the cloud who want to start/stop as many microcontainers as they currently need within 50 ms. The problem, however, is that these (artificial) bare metal start-up times and redeployment times hardly contribute to the overall success of the project. At least they influence much less than a strong framework infrastructure, strong documentation, community and strong developer skills. So it's better to look at it this way: If so far:

• You let your ORMs run rampant, generating hundreds of queries for simple workflows.
• You need endless gigabytes to run your moderately complex monolith.
• You have so much code and the complexity is so high (we're not talking about potentially slow starters like Hibernate now) that your application takes several minutes to load.

If this is the case, then adding additional mycoservice issues (network resiliency, messaging, DevOps, infrastructure) will have a much greater impact on your project than loading an empty Hello, world. And for hot redeployments during development, you might end up with solutions like JRebel or DCEVM . Let’s take a moment to quote Simon Brown again : “ If people can’t build (fast and efficient) monoliths, they will have a hard time building (fast and efficient) microservices, regardless of structure .” So choose your frameworks wisely.

Which libraries are best for synchronous Java REST calls?

On the low-level technical side, you'll likely end up with one of the following HTTP client libraries: Java's native HttpClient (since Java 11), Apache's HttpClient , or OkHttp . Note that I say "probably" here because there are other options, ranging from good old JAX-RS clients to modern WebSocket clients . In any case, the trend is towards generating an HTTP client, moving away from fiddling around with HTTP calls yourself. To do this, you need to take a look at the OpenFeign project and its documentation as a starting point for further reading.

What are the best brokers for asynchronous Java messaging?

Most likely, you will come across the popular ActiveMQ (Classic or Artemis) , RabbitMQ or Kafka .

• ActiveMQ and RabbitMQ are traditional, full-fledged message brokers. They involve the interaction of a “smart broker” and “stupid users”.
• Historically, ActiveMQ has had the benefit of easy inlining (for testing), which can be mitigated with RabbitMQ/Docker/TestContainer settings.
• Kafka cannot be called a traditional “smart” broker. Instead, it is a relatively "dumb" message store (log file) that requires smart consumers to process.

To better understand when to use RabbitMQ (or other traditional message brokers in general) or Kafka, take a look at this Pivotal post as a starting point. In general, when choosing a messaging broker, try to ignore artificial performance reasons. There was a time when teams and online communities would constantly argue about how fast RabbitMQ was and how slow ActiveMQ was. Now the same arguments are made regarding RabbitMQ, they say it works slowly with 20-30 thousand messages per second. Kafka records 100 thousand messages per second. Frankly speaking, such comparisons are like comparing warm with soft. Additionally, in both cases the throughput values ​​may be in the low to mid range of, say, Alibaba Group. However, you are unlikely to have encountered projects of this scale (millions of messages per minute) in reality. They definitely exist and they would have problems. Unlike the other 99% of “regular” Java business projects. So don't pay attention to fashion and hype. Choose wisely.

What libraries can I use to test microservices?

It depends on your stack. If you have a Spring ecosystem deployed, it would be wise to use the framework's specific tools . If JavaEE is something like Arquillian . It might be worth taking a look at Docker and the really good Testcontainers library , which helps in particular to easily and quickly set up an Oracle database for local development or integration tests. For mock testing entire HTTP servers, check out Wiremock . To test asynchronous messaging, try implementing ActiveMQ or RabbitMQ and then writing tests using Awaitility DSL . In addition, all your usual tools are used - Junit , TestNG for AssertJ and Mockito . Please note that this is not a complete list. If you don't find your favorite tool here, please post it in the comments section.

How to enable logging for all Java microservices?

Logging in the case of microservices is an interesting and rather complex topic. Instead of having one log file that you can manipulate with less or grep commands, you now have n log files, and you want them not to be too scattered. The features of the logging ecosystem are well described in this article (in English). Be sure to read it, and pay attention to the section Centralized logging from a microservices perspective . In practice, you will come across different approaches: The system administrator writes certain scripts that collect and merge log files from different servers into a single log file and put them on FTP servers for downloading. Running cat/grep/unig/sort combinations in parallel SSH sessions. This is exactly what Amazon AWS does, and you can let your manager know. Use a tool like Graylog or ELK Stack (Elasticsearch, Logstash, Kibana)

How do my microservices find each other?

Until now, we have assumed that our microservices know about each other and know the corresponding IPS. Let's talk about static configuration. So, our banking monolith [ip = 192.168.200.1] knows that it needs to talk to the risk server [ip = 192.168.200.2], which is hardcoded in the properties file. However, you can make things more dynamic:

• Use a cloud-based configuration server from which all microservices pull their configurations instead of deploying application.properties files on their microservices.
• Since your service instances can dynamically change their location, it's worth looking at services that know where your services live, what their IPs are, and how to route them.
• Now that everything is dynamic, new problems arise, such as the automatic election of a leader: who is the master who works on certain tasks, so as not to process them twice, for example? Who replaces a leader when he fails? On what basis does the replacement take place?

In general terms, this is what is called microservice orchestration and it is another bottomless topic. Libraries like Eureka or Zookeeper try to "solve" these problems by showing which services are available. On the other hand, they introduce additional complexity. Ask anyone who has ever installed ZooKeeper.

How to organize authorization and authentication using Java microservices?

This topic is also worthy of a separate story. Again, options range from hardcoded basic HTTPS authentication with custom security frameworks to running an Oauth2 installation with its own authorization server.

How can I make sure that all my environments look the same?

What is true for deployments without a microservice is also true for deployments with one. Try a combination of Docker/Testcontainers and Scripting/Ansible.

No question: briefly about YAML

Let's step away from libraries and related issues for a moment and take a quick look at Yaml. This file format is used de facto as a format for “writing configuration as code.” It is also used by simple tools like Ansible and giants like Kubernetes. To experience the pain of YAML indentation, try writing a simple Ansible file and see how much you have to edit the file before it works as expected. And this despite the format being supported by all major IDEs! After that, come back to finish reading this guide.

Yaml:
  - is:
    - so
    - great

What about distributed transactions? Performance testing? Other topics?

Maybe someday, in future editions of the manual. For now, that’s all. Stay with us!

Conceptual Issues with Microservices

Besides the specific problems of microservices in Java, there are other problems, let's say, that appear in any microservice project. They relate mostly to organization, team and management.

Frontend and Backend mismatch

Frontend and Backend mismatch is a very common problem in many microservice projects. What does it mean? Only that in the good old monoliths, web interface developers had one specific source for obtaining data. In microservice projects, front-end developers suddenly have n sources to obtain data from. Imagine that you are creating some kind of IoT (Internet of Things) microservices project in Java. Let's say you manage geodetic machines and industrial furnaces throughout Europe. And these ovens send you regular updates with their temperatures and the like. Sooner or later you might want to find ovens in the admin UI, perhaps using "furnace search" microservices. Depending on how strictly your backend counterparts apply domain-driven design or microservice laws, a “find oven” microservice may only return oven IDs and not other data such as type, model, or location. To do this, frontend developers will need to make one or n additional calls (depending on the paging implementation) in the “get furnace data” microservice with the IDs they received from the first microservice. And although this is just a simple example, albeit taken from a real (!) project, even it demonstrates the following problem: supermarkets have become extremely popular. That's because with them you don't have to go to 10 different places to buy vegetables, lemonade, frozen pizza and toilet paper. Instead, you go to one place. It's easier and faster. The same goes for front-end and microservice developers.

Management Expectations

Management is under the mistaken impression that they now need to hire an infinite number of developers for an (overarching) project, since developers can now work completely independently of each other, each on their own microservice. Only a little integration work is required at the very end (shortly before launch). In fact, this approach is extremely problematic. In the following paragraphs we will try to explain why.

“Smaller pieces” do not equal “better pieces”

It would be a big mistake to assume that code divided into 20 parts will necessarily be of higher quality than one whole piece. Even if we take quality from a purely technical standpoint, our individual services may still be running 400 Hibernate queries to select a user from the database, traversing layers of unsupported code. Once again, we return to the Simon Brown quote: if you fail to build monoliths properly, it will be difficult to build proper microservices. It is often extremely late to talk about fault tolerance in microservice projects. So much so that it is sometimes scary to see how microservices work in real projects. The reason for this is that Java developers are not always ready to study fault tolerance, networking and other related topics at the proper level. The “pieces” themselves are smaller, but the “technical parts” are larger. Imagine your microservices team is asked to write a technical microservice for logging into a database system, something like this:

@Controller
class LoginController {
    // ...
    @PostMapping("/login")
    public boolean login(String username, String password) {
        User user = userDao.findByUserName(username);
        if (user == null) {
            // обработка варианта с несуществующим пользователем
            return false;
        }
        if (!user.getPassword().equals(hashed(password))) {
            // обработка неверного пароля
            return false;
        }
        // 'Ю-ху, залогинorсь!';
        // установите cookies, делайте, что угодно
        return true;
    }
}

Now your team may decide (and perhaps even convince the business people) that this is all too simple and boring, instead of writing a login service, it’s better to write a really useful UserStateChanged microservice without any real and tangible business implications. requirements. And since some people currently treat Java like a dinosaur, let's write our UserStateChanged microservice in fashionable Erlang. And let's try using red-black trees somewhere, because Steve Yegge wrote that you have to know them inside out to apply to Google. From an integration, maintenance, and overall design perspective, this is as bad as writing layers of spaghetti code inside a single monolith. An artificial and ordinary example? This is true. However, this can happen in reality.

Fewer pieces - less understanding

Then the question naturally comes up about understanding the system as a whole, its processes and work flows, but at the same time you, as a developer, are only responsible for working on your isolated microservice [95: login-101: updateUserProfile]. It harmonizes with the previous paragraph, but depending on your organization, level of trust and communication, this can lead to a lot of confusion, shrugs, and blame if there is an accidental breakdown in the microservice chain. And there is no one who would take full responsibility for what happened. And it’s not a matter of dishonesty at all. In fact, it is very difficult to connect different parts and understand their place in the overall picture of the project.

Communications and service

The level of communication and service varies greatly depending on the size of the company. However, the general relationship is obvious: the more, the more problematic.

• Who runs microservice #47?
• Did they just deploy a new, incompatible version of a microservice? Where was this documented?
• Who do I need to talk to to request a new feature?
• Who will support that microservice in Erlang, after the only one who knew this language left the company?
• All of our microservice teams work not only in different programming languages, but also in different time zones! How do we coordinate all this correctly?

The main point is that, as with DevOps, a full-fledged approach to microservices in a large, perhaps even international company, comes with a bunch of additional communication problems. And the company must seriously prepare for this.

conclusions

After reading this article, you may think that the author is an ardent opponent of microservices. This is not entirely true - I'm basically trying to highlight points that few people pay attention to in the mad race for new technologies.

Microservices or monolith?

Using Java microservices anytime, anywhere is one extreme. The other turns out to be something like hundreds of good old Maven modules in a monolith. Your task is to find the right balance. This is especially true for new projects. There is nothing to stop you from taking a more conservative, “monolithic” approach and creating fewer good Maven modules, rather than starting with twenty cloud-ready microservices.

Microservices generate additional complexity

Keep in mind that the more microservices you have and the less really powerful DevOps you have (no, running a couple of Ansible scripts or deploying to Heroku doesn't count!), the more problems you'll have later in the process. Even just reading to the end of the section of this guide dedicated to general questions about Java microservices is quite a tedious task. Think hard about implementing solutions to all of these infrastructure problems and you'll suddenly realize that it's no longer about business programming (what you get paid to do) but rather about fixing more technology on even more technology. Shiva Prasad Reddy summed it up perfectly in his blog : “You have no idea how terrible it is when a team spends 70% of the time struggling with this modern infrastructure and only 30% of the time is left to do the actual business logic.” Shiva Prasad Reddy

Is it worth creating Java microservices?

To answer this question, I'd like to end this article with a very cheeky, Google interview-like teaser. If you know the answer to this question from experience, even if it doesn't seem to have anything to do with microservices, you may be ready for a microservices approach.

Scenario

Imagine you have a Java monolith running alone on the smallest Hetzner dedicated server . The same applies to your database server, it is also running on a similar Hetzner machine . And let's also assume that your Java monolith can handle workflows, say user registration, and you're not creating hundreds of database queries per workflow, but a more reasonable number (<10).

Question

How many database connections should your Java monolith (connection pool) open on your database server? Why is that? How many concurrent active users do you think your monolith can (approximately) scale?

Answer

Leave your answer to these questions in the comments section. I look forward to all the answers. Now make up your mind. If you read to the very end, we are very grateful to you!