Java Microservices Guide. Part 3: general questions

Translating and Adapting Java Microservices: A Practical Guide . Previous parts of the guide:

• the basics of microservices and their architecture ;
• deploying and testing microservices .

Let's take a look at the inherent problems of Java microservices from abstract things to specific libraries.

How to make a Java microservice resilient?

Recall that when you create microservices, you are essentially changing JVM method calls to synchronous HTTP calls or asynchronous messaging. While the execution of a method call is mostly guaranteed (with the exception of an unexpected JVM termination), a network call is unreliable by default. It may or may not work for various reasons: the network is congested, a new firewall rule has been implemented, and so on. To see how this matters, let's take a look at the BillingService example.

HTTP/REST Resiliency Patterns

Let's say customers can buy e-books from your company's website. To do this, you've just implemented a billing microservice that can call your online store to generate actual PDF invoices. We'll now make this call synchronously over HTTP (although it makes more sense to call this service asynchronously, since PDF generation doesn't have to be instantaneous from the user's point of view. We'll use the same example in the next section and look at the differences).

@Service
class BillingService {

    @Autowired
    private HttpClient client;

     public void bill(User user, Plan plan) {
        Invoice invoice = createInvoice(user, plan);
        httpClient.send(invoiceRequest(user.getEmail(), invoice), responseHandler());
        // ...
    }
}

To summarize, here are three possible outcomes of this HTTP call.

• OK: the call went through, the account was created successfully.
• DELAYED: The call went through, but it took too long to complete.
• ERROR. The call did not go through, you may have sent an incompatible request, or the system may not have worked.

Any program is expected to handle error situations, not just successful ones. The same applies to microservices. Even if you have to put in extra effort to ensure that all deployed versions of the API are compatible, once you start with deployments and releases of individual microservices. An interesting case worth paying attention to is the delay case. For example, the respondent's microservice hard drive is full, and instead of 50ms, it takes 10 seconds to respond. It gets even more interesting when you're under a certain load, so that the unresponsiveness of your BillingService starts cascading through your system. As an illustrative example, imagine a kitchen slowly starting a "block" of all the waiters in a restaurant. This section obviously cannot provide an exhaustive overview of the topic of microservice resilience, but serves as a reminder to developers that this is actually something to be addressed and not ignored prior to your first release (which, in experience, happens more often than follows). A popular library that helps you think about latency and fault tolerance is Netflix's Hystrix.

Messaging Resilience Patterns (Fault-tolerant messaging patterns)

Let's take a closer look at asynchronous communication. Our BillingService program might now look something like this, assuming we're using Spring and RabbitMQ for messaging. To create an account, we now send a message to our RabbitMQ message broker, where there are several workers waiting for new messages. These workers create invoices in PDF format and send them to the appropriate users.

@Service
class BillingService {

    @Autowired
    private RabbitTemplate rabbitTemplate;

     public void bill(User user, Plan plan) {
        Invoice invoice = createInvoice(user, plan);
        // преобразует счет, например, в json и использует его How тело messages
        rabbitTemplate.convertAndSend(exchange, routingkey, invoice);
        // ...
    }
}

Potential errors now look a little different, as you no longer get immediate OK or ERROR responses like you did with a synchronous HTTP connection. Instead, we could have three potential wrong developments that could raise the following questions:

1. Was my message delivered and used by a worker? Or is it lost? (The user does not receive an invoice).
2. Was my message delivered only once? Or delivered more than once and only processed once? (User will receive multiple invoices).
3. Configuration: From "Did I use the correct routing keys/names for the exchange" to "Is my message broker properly configured and maintained, or are its queues full?" (The user does not receive an invoice).

A detailed description of each individual asynchronous microservice resiliency pattern is beyond the scope of this tutorial. However, there are pointers in the right direction. Especially since they will depend on the messaging technology. Examples:

• If you're using JMS implementations like ActiveMQ, you can trade speed for guaranteed two-phase (XA) commits.
• If you're using RabbitMQ, read this guide first, and then think carefully about confirmations, fault tolerance, and message reliability in general.
• Perhaps someone is well versed in configuring Active or RabbitMQ servers, especially in combination with clustering and Docker (anyone? ;) )

Which framework would be the best solution for Java microservices?

On the one hand, a very popular option like Spring Boot can be installed . It makes it very easy to create .jar files, comes with a built-in web server like Tomcat or Jetty, and can be run quickly and anywhere. Ideal for building microservice applications. Not so long ago, a couple of specialized Kubernetes or GraalVM microservice frameworks appeared , partly inspired by reactive programming. Here are some more interesting contenders: Quarkus , Micronaut , Vert.x , Helidon. In the end, you'll have to choose for yourself, but we can give you a couple of recommendations, perhaps not quite standard: With the exception of Spring Boot, all microservice platforms are usually positioned as incredibly fast, with almost instant startup, low memory usage, the ability to scale up to infinity. Marketing materials usually include impressive graphics that show the platform in a favorable light next to the Spring Boot behemoth or with each other. This, in theory, spares the nerves of developers who support legacy projects, which sometimes take several minutes to load. Or developers working in the cloud who want to start/stop as many microcontainers as they currently need within 50ms. The problem, however, is that such (artificial) bare metal start times and redeployment times hardly affect the overall success of the project. At least the impact is much less than a strong framework infrastructure, strong documentation, community, and strong developer skills. So it's better to look at it this way: If so far:

• You let your ORM run wild generating hundreds of requests for simple workflows.
• You need infinite gigabytes to run your moderately complex monolith.
• You have so much code and so much complexity (now we're not talking about potentially slow starters like Hibernate) that it takes a few minutes for your application to load.

If this is the case, then adding additional mycoservice concerns (failover networking, messaging, DevOps, infrastructure) will have a much larger impact on your project than loading an empty Hello, world. And for hot redeployments during development, you might end up using solutions like JRebel or DCEVM . Let's not be too lazy to quote Simon Brown again : “ if people cannot create (fast and efficient) monoliths, it will be difficult for them to create (fast and efficient) microservices regardless of the structure .” So choose your frameworks wisely.

What libraries are best for synchronous Java REST calls?

On the low-level technical side, you'll probably end up with one of the following HTTP client libraries: Java's native HttpClient (as of Java 11), Apache's HttpClient , or OkHttp . Note that I say "probably" here because there are other options, ranging from good old JAX-RS clients to modern WebSocket clients . In any case, there is a tendency to generate an HTTP client, away from fiddling with HTTP calls yourself. To do this, you need to take a look at the OpenFeign project and its documentation as a starting point for further reading.

Which brokers are the best for Java asynchronous messaging?

Most likely you will come across the popular ActiveMQ (Classic or Artemis) , RabbitMQ or Kafka .

• ActiveMQ and RabbitMQ are traditional, full fledged message brokers. They involve the interaction of a "smart broker" and "stupid users".
• Historically, ActiveMQ has had the advantage of simple inlining (for testing), which can be mitigated through RabbitMQ/Docker/TestContainer settings.
• Kafka is not a traditional smart broker. Instead, it is a relatively "stupid" message store (log file) that needs smart consumers to process.

To better understand when to use RabbitMQ (or other traditional message brokers in general) or Kafka, take a look at this post on Pivotal(in English) as a starting point. In general, when choosing a messaging broker, try to ignore artificial performance reasons. There was a time when teams and online communities were constantly arguing about how fast RabbitMQ was and how slow ActiveMQ was. Now the same arguments are being made for RabbitMQ, saying it is slow with 20-30 thousand messages per second. Kafka has 100,000 messages per second. Frankly speaking, such comparisons are like comparing warm with soft. In addition, in both cases, the throughput values ​​may be at the low or average level, say, for the Alibaba Group. However, you are unlikely to come across projects of this magnitude (millions of messages per minute) in reality. They definitely exist and they would be in trouble. Unlike the other 99% of “regular” Java business projects. So don't pay attention to fashion and hype. Choose wisely.

What libraries can I use to test microservices?

It depends on your stack. If you have a Spring ecosystem deployed, it will be wise to use the special tools of this framework . If JavaEE is something like Arquillian . It might be worth taking a look at Docker and the really good Testcontainers library , which helps in particular to easily and quickly set up an Oracle database for local development or integration tests. For mock tests of entire HTTP servers, check out Wiremock . For testing asynchronous messaging, try implementing ActiveMQ or RabbitMQ and then writing tests using the Awaitility DSL . In addition, all your usual tools are applied - Junit , TestNGfor AssertJ and Mockito . Please note that this is not a complete list. If you don't find your favorite tool here, post it in the comments section.

How to enable logging for all Java microservices?

Logging in the case of microservices is an interesting and rather complex topic. Instead of having one log file that you can work with less or grep commands, you now have n log files, and it is desirable that they are not too scattered. The features of the logging ecosystem are well described in this article (in English). Be sure to read it, and pay attention to the section Centralized logging from a microservices perspective. In practice, you will come across different approaches: The system administrator writes certain scripts that collect and combine log files from various servers into one log file and put them on FTP servers for download. Running cat/grep/unig/sort combinations in parallel SSH sessions. This is exactly what Amazon AWS does, which you can report to your manager. Use a tool like Graylog or ELK Stack (Elasticsearch, Logstash, Kibana)

How do my microservices find each other?

Until now, we have assumed that our microservices know about each other, know the corresponding IPS. Let's talk about static tuning. So, our banking monolith [ip = 192.168.200.1] knows that it needs to talk to the risk server [ip = 192.168.200.2], which is hardcoded in the properties file. However, you can make everything more dynamic:

• Use a cloud-based configuration server from which all microservices pull their configurations instead of deploying application.properties files to their microservices.
• Because your service instances can change location dynamically, it's worth looking into services that know where your services live, what their IPs are, and how to route them.
• Now that everything is dynamic, new problems appear, such as automatic leader election: who is the master who works on certain tasks so that, for example, they do not process them twice? Who replaces the leader when he fails? What is the principle of replacement?

In general terms, this is what is called microservice orchestration and it is another bottomless topic. Libraries like Eureka or Zookeeper try to "solve" these problems by showing what services are available. On the other hand, they introduce additional complexity. Ask anyone who has ever installed ZooKeeper.

How to organize authorization and authentication using Java microservices?

This topic also deserves a separate story. Again, options range from hardcoded HTTPS basic authentication with self-written security frameworks to running an Oauth2 setup with your own authorization server.

How do I make sure all my environments look the same?

What is true for deployments without a microservice is also true for deployments with one. Try a combination of Docker/Testcontainers as well as Scripting/Ansible.

Not a Question: Briefly About YAML

Let's step away from libraries and related issues for a moment and take a quick look at Yaml. This file format is used de facto as the format for "writing configuration as code". Simple tools like Ansible and giants like Kubernetes also use it. To experience the pain of YAML indentation, try writing a simple Ansible file and see how much you have to edit the file before it works properly. And this is despite the support of the format by all major IDEs! After that, come back to finish reading this guide.

Yaml:
  - is:
    - so
    - great

What about distributed transactions? Performance testing? Other topics?

Maybe someday, in the next editions of the manual. For now, that's all. Stay with us!

Conceptual problems of microservices

In addition to the specific problems of microservices in Java, there are other problems, say, those that appear in any microservice project. They are mostly about organization, team and management.

Mismatch Frontend and Backend

Mismatch between Frontend and Backend is a very common problem in many microservice projects. What does she mean? Only that in the good old monoliths, the developers of the web interface had one specific source for obtaining data. In microservice projects, web interface developers suddenly have n sources for getting data. Imagine that you are creating some kind of IoT (Internet of Things) microservices project in Java. Let's say you are in charge of surveying machines, industrial furnaces throughout Europe. And these ovens send you regular updates with their temperatures and stuff like that. Sooner or later, you may want to find stoves in the admin UI, perhaps using "stove finder" microservices. Depending on how strictly your backend colleagues applydomain-driven design or the laws of microservices, a “find oven” microservice can only return oven IDs, not other data such as type, model, or location. To do this, front-end developers will need to make one or n additional calls (depending on the paging implementation) to the “get oven data” microservice with the IDs they received from the first microservice. And although this is just a simple example, albeit taken from a real (!) project, even it demonstrates the following problem: supermarkets have become extremely popular. That's because with them you don't have to go to 10 different places to buy vegetables, lemonade, frozen pizza and toilet paper. Instead, you go to one place. It's easier and faster. The same goes for front-end and microservice developers.

Leadership expectations

Management is under the erroneous impression that it is now necessary to hire an infinite number of developers in a (comprehensive) project, since developers can now work completely independently from each other, each on their own microservice. At the very end, only a little integration work is required (shortly before launch). In fact, this approach is extremely problematic. In the following paragraphs we will try to explain why.

“Smaller pieces” does not equal “better pieces”

It would be a big mistake to assume that code divided into 20 parts will necessarily be better than one whole piece. Even if we take the quality from a purely technical point of view, our individual services can still perform 400 Hibernate requests to select a user from the database, traversing layers of unmaintained code. Once again, we return to Simon Brown's quote: if you can't build monoliths properly, it will be difficult to create proper microservices. Often, fault tolerance in microservice projects is extremely out of time. So much so that sometimes it’s scary to watch how microservices work in real projects. The reason for this lies in the fact that Java developers are not always ready to study fault tolerance, networks and other related topics at the proper level. The “pieces” themselves are smaller, but there are more “technical parts”. Imagine that your microservice team is asked to write a technical database login microservice, something like this:

@Controller
class LoginController {
    // ...
    @PostMapping("/login")
    public boolean login(String username, String password) {
        User user = userDao.findByUserName(username);
        if (user == null) {
            // обработка варианта с несуществующим пользователем
            return false;
        }
        if (!user.getPassword().equals(hashed(password))) {
            // обработка неверного пароля
            return false;
        }
        // 'Ю-ху, залогинorсь!';
        // установите cookies, делайте, что угодно
        return true;
    }
}

Now your team can decide (and maybe even convince business people) that this is all too simple and boring, instead of a login service, write a really useful UserStateChanged microservice (user state change) without any real and tangible business requirements. And since some people now treat Java like a dinosaur, let's write our UserStateChanged microservice in fancy Erlang. And let's try to use red-black trees somewhere, because Steve Yegge wrote that you have to know them from the inside in order to apply to Google. In terms of integration, maintenance, and overall design, this is about as bad as writing layers of spaghetti code inside one monolith. An artificial and ordinary example? This is true. However, this may be the case in reality.

Less pieces - less understanding

Then the question naturally arises of understanding the system as a whole, its processes and workflows, but at the same time, you, as a developer, are only responsible for working on your isolated microservice [95: login-101: updateUserProfile]. It harmonizes with the previous paragraph, but depending on your organization, level of trust and communication, this can lead to a lot of bewilderment, shrug, blame in case of an accidental breakdown in the microservice chain. And there is no one who would take full responsibility for what happened. And it's not all about dishonesty. In fact, it is very difficult to connect different details and understand their place in the overall picture of the project.

Communications and service

The level of communication and service strongly depends on the size of the company. Nevertheless, the general dependence is obvious: the more, the more problematic.

• Who works on microservice #47?
• Did they just deploy a new incompatible version of the microservice? Where has it been documented?
• Who do I need to talk to to request a new feature?
• Who will maintain that microservice in Erlang, after the only one who knew this language left the company?
• All our microservice teams work not only in different programming languages, but also in different time zones! How do we coordinate all this correctly?

The bottom line is that, just like DevOps, a full microservices approach in a large, perhaps even international, company comes with a bunch of additional communication challenges. And the company should seriously prepare for this.

conclusions

After reading this article, you may decide that the author is an ardent opponent of microservices. This is not entirely true - I'm mainly trying to highlight the points that few people pay attention to in the crazy race for new technologies.

Microservices or monolith?

Using Java microservices anytime, anywhere is one extreme. The other turns out to be something like hundreds of good old Maven modules in a monolith. Your job is to find the right balance. This is especially true for new projects. Here, nothing will stop you from taking a more conservative, “monolithic” approach and creating fewer good Maven modules, instead of starting with twenty cloud-ready microservices.

Microservices generate additional complexity

Keep in mind that the more microservices you have and the fewer truly powerful DevOps you have (no, running a couple of Ansible scripts or deploying to Heroku does not count!), the more problems you will have later in the work. Even just reading through to the end of the section of this guide on general questions about Java microservices is a rather tedious task. Think hard about implementing solutions for all these infrastructure tasks and suddenly you realize that it's not about business programming anymore (which you get paid to do), but rather fixing more technology on even more technology. Shiva Prasad Reddy summed it up nicely in his blog : “You have no idea how terrible it is when a team is 70% of the time struggling with this modern infrastructure and only 30% of the time is left for real business logic.” Shiva Prasad Reddy

Is it worth building Java microservices?

To answer that question, I'd like to end this article with a very cheeky, Google interview-like teaser. If you know the answer to this question from experience, even if it doesn't seem to have anything to do with microservices, you might be ready for the microservice approach.

Scenario

Imagine that you have a Java monolith running alone on the smallest Hetzner dedicated server . The same applies to your database server, it also runs on a similar Hetzner machine . And let's also assume that your Java monolith can handle workflows, say user registration, and you're not making hundreds of database queries per workflow, but a more reasonable number (<10).

Question

How many database connections should your Java monolith (connection pool) open on your database server? Why is that? How many concurrent active users do you think your monolith can (approximately) scale?

Answer

Leave your answer to these questions in the comments section. I look forward to all responses. Now make up your mind. If you have read to the very end, we are very grateful to you!