Telegram-bot as the first project and its importance for professional growth from personal experience

Greetings to all! Tell us about yourself. I am 24 years old, graduated from a technical university last year and still have no work experience. Looking ahead, I want to say that initially, in the laid down plan (drawn up in the fall of 2019), I planned to go to work in March-April 2020, but, unfortunately, quarantine intervened, so I postponed everything until mid-summer and in the future I hope to write my own success story. I have never been drawn to programming. Programming was taught enough at the university, but this craft could not interest me then. There were also procedural languages ​​(C), a one-year OOP course (Java), databases, even assembler and C++. But what to hide, I was indifferent to studying in general, since most of the disciplines taught seemed useless to me, suitable only for a reporting sheet (in principle, this is true). After graduating from the university, I had to make a decision: I didn’t acquire any skills, but I had to work. I had to think about self-education (oh, I already missed at least 2 full years, sitting idly by) and the choice fell on Java by itself, because on the OOP course at the university, one of the guys advised the codegym course, and he, as you know , dedicated specifically to the Java language. Interested in the course. Yes, I didn’t like programming then, because I immediately gave up this business when I met some kind of complexity, and there are more than enough difficulties in programming. But at the same time, I felt like I wanted to write code, which is why I eventually decided to get into programming. I will briefly tell you about my experience on codegym. I started in August 2019, immediately bought a subscription for a month, but at level 7 I realized that the tasks are difficult. Postponed the course, picked up Schildt. So in parallel, I took the course for 3 months. I reached level 20 (this is my second account), almost completely read Schildt, then I got tired of the local tasks, in which I stopped seeing practical benefits for myself. I went to codewars, leetcode, started watching video courses. By the way, in 3 months I went from "Oh no, what is an array? How to work with it and why is it so scary" ? to a detailed study of the source code of collection classes (ArrayList, HashMap, etc.). Based on personal experience, I’ll tell beginners: the main thing here is to overcome the feeling that arises if you don’t understand anything and can’t decide anything. When it arises, you just want to quit everything and it seems that you are too stupid for this business. If you overcome such moments in yourself and mentally relax, then success will come. I think that many do not cope with this, so they quickly give up such undertakings. As a result, in December 2019 I thought about my project. I decided to choose a Telegram bot, but there was no idea. At the same time, one friend needed functionality for his group in a telegram, which he would like to automate. He was just aware that I was studying programming in depth and offered me a project. Me for experience and future resume, him for the development of the group. I even take the liberty of quoting his idea:Recently, I wanted to order software from a programmer, which would upload files to the selected Cloud via direct links. This is interesting, since there are no analogues. And it's just very convenient. The bottom line: you copy the link, paste it into the window and select the desired Cloud (GDrive, Mail, Yandex Disk, etc.), at one time the software does everything on the server side and the user does not need to download anything to his machine (especially cool when you have assembly on SSD drives). We thought to do it in the web interface, so that it could be launched both from phones and from the desktop ... In principle, it can be implemented through the application, and not through the web interface. Are you capable of that?Thus, with each request, the library can be expanded due to the efforts of users. In the future, you can get information about the game in a convenient form without going to Google Play. You just write /library here_name_of_game and get everything you need. But there are several difficulties, which I will talk about later. At first, I was moving slowly, as I began to take two SQL courses in parallel. Tritely could not understand how the bot works in general, and how to process requests. I met a friend who was also interested in working on the project. The first version of the bot was ready in about a month, but disagreements arose with a friend (on my part). I took care of the part of the bot that is responsible for parsing, and he directly worked on requests to the bot and their processing. For some reason, he began to complicate the bot, introduce some kind of authorization, invent admins, add unnecessary functionality, plus I didn't quite like his coding style. In my opinion, this was not needed in the information bot. So I decided that I would write a bot from scratch myself with the functionality I needed. Now I’ll tell you what the bot actually does (using an example from the project code). I will attach the full code of the project at the end of the article and, unfortunately, I cannot physically comment on it in full. Any user message sent to the bot is an object of the Update class. It contains a lot of information (message id, chat id, unique user id, etc.). There are several types of update: it can be a text message, it can be a response from the telegram keyboard (callback), a photo, audio, etc. So that the user does not particularly indulge, I process only text requests and callbacks from the keyboard. If the user sends a photo, the bot will notify him that he does not intend to do anything with it. In the main bot class, in the onUpdateReceived method, the bot receives an update.

@Override
    public void onUpdateReceived(Update update) {
        UpdatesReceiver.handleUpdates(update);
    }

which I pass to the handler (own UpdatesReceiver class):

public static void handleUpdates(Update update) {
        ...
        if (update.hasMessage() && update.getMessage().hasText()){
            log.info("[Update (id {}) типа \"Текстовое сообщение\"]", update.getUpdateId());
            new TextMessageHandler(update, replyGenerator).handleTextMessage();
        }
        else if (update.hasCallbackQuery()) {
            //логгирование
            new CallbackQueryHandler(update, replyGenerator).handleCallBackQuery();
        }
        else {
           //логгирование
            replyGenerator.sendTextMessage(update.getMessage().getChatId(), "Я могу принимать только текстовые messages!");
        }
    }

UpdatesReceiver is a central handler that, depending on the type of update, transfers control to another specialized handler: TextMessageHandler or CallbackQueryHandler, to whose constructors I pass update further along the chain. Update is the most important thing when working with a bot and should not be lost, because with the help of the information stored in the update, we will find out to which user and to which chat the answer should be sent. To generate responses to the user, I wrote a separate class. It can send normal text message, inline keyboard message, picture message and reply keyboard message. The inline keyboard looks like this: It defines buttons, by clicking on which, the user sends a callback to the server, which can be processed in much the same way as regular messages. To "maintain" it, you need your own handler. We set an action for each button, which is then written to the Update object. Those. for the "Price" button, we set the description "/price" for the callback, which we can later get from the update. Further in a separate class, I can already process this callback:

public void handleCallBackQuery() {
  String call_data = update.getCallbackQuery().getData();
  long message_id = update.getCallbackQuery().getMessage().getMessageId();
  long chat_id = update.getCallbackQuery().getMessage().getChatId();
    switch (call_date){
      case "/price" :
        //тут что-то сделать
        break;
...

The Reply keyboard looks like this: And in fact, it replaces the user's typing. By clicking on the "Library" button, you will quickly send a "Library" message to the bot. For each type of keyboard, I wrote my own class, implementing the Builder pattern: inline and reply . As a result, you can essentially "draw" the desired keyboard, depending on the requirements. This is terribly convenient, since keyboards can be different, but the principle remains the same. Here is an intuitive method to send a message with an inline keyboard:

public synchronized void sendInlineKeyboardMessage(long chat_id, String gameTitle) {
        SendMessage keyboard = InlineKeyboardMarkupBuilder.create(chat_id)
                .setText("Вы может узнать следующую информацию об игре " + gameTitle)
                .row()
                .button("Стоимость " + "\uD83D\uDCB0", "/price " + gameTitle)
                .button("Обновлено " + "\uD83D\uDDD3", "/updated " + gameTitle)
                .button("Версия " + "\uD83D\uDEE0", "/version " + gameTitle)
                .endRow()
                .row()
                .button("Требования " + "\uD83D\uDCF5", "/requirements " + gameTitle)
                .button("Покупки " + "\uD83D\uDED2", "/iap " + gameTitle)
                .button("Размер " + "\uD83D\uDD0E", "/size " + gameTitle)
                .endRow()
                .row()
                .button("Получить всю информацию об игре" + "\uD83D\uDD79", "/all " + gameTitle)
                .endRow()
                .row()
                .button("Скрыть клавиатуру", "close")
                .endRow()
                .build();
        try {
            execute(keyboard);
        } catch (TelegramApiException e) {
            log.error("[Не удалось отправить сообщение с -inline- клавиатурой]: {}", e.getMessage());
        }
    }

To give the bot strict functionality, special commands were invented through the slash symbol: /library, /help, /game, etc. Otherwise, we would have to process any garbage that the user could write. Actually, MessageHandler was written for this:

if (message.equals(ChatCommands.START.getDescription())) {
     replyGenerator.sendTextMessage(chat_id, new StartMessageHandler().reply());
     replyGenerator.sendReplyKeyboardMessage(chat_id);
}
else if (message.equals(ChatCommands.HELP.getDescription())
             || message.equalsIgnoreCase("Помощь")) {
      replyGenerator.sendTextMessage(chat_id, new HelpMessageHandler().reply());
}
 ...

Thus, depending on what command you send to the bot, a special handler will be included in the work. Let's go further and consider the work of the parser and the library. If you send a link to a game in the Google Play store to the bot, then a special handler will automatically work . In response, the user will receive information about the game in the following form: Along with this, a method will be called that will try to add the game to the bot's library (first to the local map, then to the -> json file). If the game is already in the library, then a check will be made (as in a regular hashmap), and if the field data (for example, the version number has changed), then the game in the library will be overwritten. If no changes are detected, then no records will be made. If there was no game in the library at all, then it is first written to the local map (object like tyk ), and then written to the json file, since if the application unexpectedly closes on the server, the data will be lost, and using the file they can always be read. Actually, when the program starts, the library is always loaded for the first time from a file from a static block:

static {
        TypeFactory typeFactory = mapper.getTypeFactory();
        MapType mapType = typeFactory.constructMapType(ConcurrentSkipListMap.class, String.class, GooglePlayGame.class);

        try {
            Path path = Paths.get(LIBRARY_PATH);
            if (!Files.exists(path)) {
                Files.createDirectories(path.getParent());
                Files.createFile(path);
                log.info("[Файл библиотеки создан]");
            }
            else {
                ConcurrentMap<string, googleplaygame=""> temporary = mapper.readValue(new File(LIBRARY_PATH), mapType);
                games.putAll(temporary);
                log.info("[Количество игр в загруженной библиотеке] = " + games.size());
            }
        }
        catch (IOException e) {
            log.error("[Ошибка при чтении/записи file] {}", e.getMessage());
        }
    }

Here you additionally have to read the data from the file into a temporary map, which is then "copied" into a full-fledged one in order to maintain case insensitivity when searching for a game in the file (by writing tITan QuEST, the bot will still find the Titan Quest game in the library). It was not possible to find another solution, these are the features of deserialization using Jackson. So, with each request by reference, the game is added to the library if possible, and the library is thereby expanded. Further, information about a particular game can be obtained using the /library Game_Name command.You can find out both a specific parameter (for example, the current version), and all parameters at once. This is implemented using the inline keyboard, which was discussed earlier. In the course of work, he also applied the skills acquired here in the course of solving problems. For example, a list of names of random games in the library (the option is available with the /library command):

private String getRandomTitles(){
        if (LibraryService.getLibrary().size() < 10){
            return String.join("\n", LibraryService.getLibrary().keySet());
        }
        List<string> keys = new ArrayList<>(LibraryService.getLibrary().keySet());
        Collections.shuffle(keys);
        List<string> randomKeys = keys.subList(0, 10);
        return String.join("\n", randomKeys);
    }

How does the bot handle links? It checks if they belong to Google Play (host, protocol, port):

private static class GooglePlayCorrectURL {

        private static final String VALID_HOST = "play.google.com";

        private static final String VALID_PROTOCOL = "https";

        private static final int VALID_PORT = -1;

        private static boolean isLinkValid(URI link) {
            return (isHostExist(link) && isProtocolExist(link) && link.getPort() == VALID_PORT);
        }

        private static boolean isProtocolExist(URI link) {
            if (link.getScheme() != null) {
                return link.getScheme().equals(VALID_PROTOCOL);
            }
            else {
                return false;
            }
        }

        private static boolean isHostExist(URI link) {
            if (link.getHost() != null) {
                return link.getHost().equals(VALID_HOST);
            }
            else {
                return false;
            }
        }

If everything is in order, then the bot connects via a link using the Jsoup library, which allows you to get the HTML code of the page, which is subject to further analysis and parsing. It will not work to deceive the bot with an incorrect or harmful link.

if (GooglePlayCorrectURL.isLinkValid(link)){
     if (!link.getPath().contains("apps")){
         throw new InvalidGooglePlayLinkException("К сожалению, бот работает исключительно с играми. Введите другую ссылку.");
     }
     URL = forceToRusLocalization(URL);
     document = Jsoup.connect(URL).get();
 }
     else {
         throw new NotGooglePlayLinkException();
      }
...

Here I had to solve the problem with regional settings. The bot connects to the Google Play store from a server located in Europe, so the page in the Google Play store opens in the appropriate language. I had to write a crutch that forcibly carries out a "redirect" to the Russian version of the page (the project was still aimed at our audience). To do this, at the end of the link, you need to carefully add the hl: &hl=ru parameter in the GET request to the Google Play server .

private String forceToRusLocalization(String URL) {
        if (URL.endsWith("&hl=ru")){
            return URL;
        }
        else {
            if (URL.contains("&hl=")){
                URL = URL.replace(
                        URL.substring(URL.length()-"&hl=ru".length()), "&hl=ru");
            }
            else {
                URL += "&hl=ru";
            }
        }
        return URL;
    }

After a successful connection, we get an HTML document ready for parsing and parsing, but this is already beyond the scope of this article. parser code here. The parser itself gets the necessary information and creates an object with the game, which is later added to the library if necessary. <h2>Summarize</h2>The bot supports several commands that contain certain functionality. It receives messages from the user and matches them with its commands. If it's a link or /game + link, it checks if the link belongs to Google Play. If the link is valid, it connects via Jsoup and gets an HTML document. This document is parsed based on the written parser. The necessary information about the game is extracted from the document, and then the object with the game is filled with this data. Next, the object with the game is placed in the local storage (if the game is not there yet) and immediately written to a file to avoid data loss. The game recorded in the library (the name of the game is the key for the map, the object with the game is the value for the map) can be obtained by the command /library Game_name. If the specified game is found in the bot's library, the user will be returned an inline keyboard, with which he can get information about the game. If the game is not found, you must either make sure the name is spelled correctly (it must match the name of the game in the Google Play store, except for case), or add the game to the library by sending the bot a link to the game. I deployed the bot on heroku, and for those who plan to write their own bot and host it for free on heroku in the future, I will give a couple of recommendations for solving the difficulties that you may encounter (because I encountered them myself). Unfortunately, due to the nature of heroku, the bot's library is constantly "reset" every 24 hours. My plan doesn't support file storage on heroku servers, so it just pulls my games file from github. There were several solutions: use the database, or look for another server that would store this game file. I decided not to do anything for now, because in fact the bot is not so useful. I needed it rather to get a full experience, which I basically achieved. So here are the recommendations for heroku: I needed it rather to get a full experience, which I basically achieved. So here are the recommendations for heroku: I needed it rather to get a full experience, which I basically achieved. So here are the recommendations for heroku:

1. You will most likely need to sign up for heroku using a VPN if you live in Russia.

2. At the root of the project, you need to put a file without an extension called Procfile. Its content should be like this: https://github.com/miroha/Telegram-Bot/blob/master/Procfile

3. In pom.xml, add the following lines following the pattern , where in the mainClass tag specify the path to the class that contains the main method: bot.BotApplication (if the BotApplication class is in the bot folder).

4. Do not build any project using mvn package commands, etc., heroku will build everything for you.

5. It is advisable to add gitignore to the project, for example this:

   # Log file
   *.log
   
   # Compiled resources
   target
   
   # Tests
   test
   
   # IDEA files
   .idea
   *.iml

6. Actually upload the project to github, and then connect the repository to heroku (or use other methods, there are 3 of them, if I'm not mistaken).

7. If the download was successful ("Build succeeded"), be sure to go to Configure Dynos:

   and toggle the slider, and then make sure it's ON (due to the fact that I didn't do this, my bot didn't work and I racked my brains for a couple of days and made a lot of unnecessary gestures).

8. Hide the bot token on github. To do this, you need to get the token from the environment variable:

   public class Bot extends TelegramLongPollingBot {
   
       private static final String BOT_TOKEN = System.getenv("TOKEN");
   
       @Override
       public String getBotToken() {
           return BOT_TOKEN;
       }
   ...
   }

   And then after deploying the bot, set this variable in the heroku dashboard in the Settings tab (to the right of TOKEN there will be a VALUE field, copy your bot token there):

In total, for 2 months of working on my own project, I:

• received a fully working project written in Java;
• learned to work with a third-party API (Telegram Bot API);
• in practice, delved into serialization, worked a lot with JSON and the Jackson library (initially I used GSON, but there were problems with it);
• strengthened my skills when working with files, got acquainted with Java NIO;
• learned to work with configuration .xml files and accustomed myself to logging;
• improved command of the development environment (IDEA);
• learned to work with git and learned the value of gitignore;
• gained skills in parsing web pages (Jsoup library);
• studied and used several design patterns;
• developed a sense and desire to improve the code (refactoring);
• learned to find solutions on the net and not be shy to ask questions that could not be answered.

I don’t know how useful or useless the bot turned out to be, how beautiful / ugly the code was, but the experience I got was definitely worth it. I have a sense of responsibility for my project. Every now and then I want to improve it, add something new. When I was able to run it and make sure that everything works the way I wanted, I experienced a real buzz. Isn't that the main thing? Enjoy what you do and enjoy every working line of code like the last chocolate bar. Therefore, if you are learning programming, then my advice to you is: do not stay here until level 40, but start your own project as early as possible. If anyone is interested, the source code of the project is here (rewritten for Spring): https://github.com/miroha/GooglePlayGames-TelegramBot For the last two months I have hardly studied new material, as it seems to me that I have reached a dead end. Without work, I don’t see where to develop, except to learn the Spring Framework, which is what I plan to do in the next month. And then I'll try to "rewrite" the bot using this framework. Ready to answer any questions. :) All success! UPDATE dated 07/07/2020 The repository with a pure Java bot was lost (I deleted it, the copy remained on another local machine), but I downloaded a rewritten bot for Spring Boot: https://github.com/miroha/GooglePlayGames-TelegramBot