Update statistics for admin - "Java project from A to Z"

Hello everyone, dear friends. In the last post we added admin commands, and today we'll talk about how to expand our statistics. Also, since the system is already essentially ready for MVP, we will do a little refactoring.

Who is this article for?

The article is intended for those who have already read the entire series before. For those who are here for the first time - here is the beginning . Who can master this series? Yes, almost any person who understands Java Core. Everything else (as it seems to me) I give in a series of articles. For those who were waiting for the writing of the entire project in order to start figuring it out right away and not wait for new articles, you can already start, everything is practically done.

Expanding statistics for the admin

This stage consists of two steps: planning and implementation. Let's get started.

Planning updated bot statistics

Before taking on the functionality, it is worth considering what we want. What statistics would be interesting for the admin to track the work of the bot? At first, in order to find out the opinion of people, I created a post in the telegram channel . No proposals were received: the only suggestion was to leave everything as it is, because the idea is shown, but how to implement it is up to everyone to decide for themselves. I agree, and you need to have your own opinion, so I decided to highlight the data that are of interest to me:

• you need to know the number of active users of the bot (this has already been done);
• you need to know the number of inactive users - let's add this. I think this is very useful, because it will be clear what is the proportion of users who refuse to use the bot among all users;
• the number of active users in each active group;
• average number of subscriptions per user.

This data needs to be tracked every day, or every two days, for example. And based on this data, draw statistics for the week / month / quarter. This will be a separate job that will run once a day. Also, based on this data, you can track the change in active users over time. Yes, such statistics would be useful for the admin and more understandable.

Implementing the selected statistics

I don’t think that within the framework of this article we will really have time to implement everything, but I’m sure that we will need all of it. Based on the ideas above, we will make a DTO class with the fields that we want to receive:

package com.github.codegymcommunity.jrtb.dto;

import lombok.Data;
import lombok.EqualsAndHashCode;

import java.util.List;

/**
* DTO for getting bot statistics.
*/
@Data
@EqualsAndHashCode
public class StatisticDTO {
   private final int activeUserCount;
   private final int inactiveUserCount;
   private final List<GroupStatDTO> groupStatDTOs;
   private final double averageGroupCountByUser;
}

and GroupStatDto

package com.github.codegymcommunity.jrtb.dto;

import lombok.Data;
import lombok.EqualsAndHashCode;

/**
* DTO for showing group id and title without data
*/
@Data

@EqualsAndHashCode(exclude = {"title", "activeUserCount"})
public class GroupStatDTO {

   private final Integer id;
   private final String title;
   private final Integer activeUserCount;
}

We created it in the dto package next to service, bot, command and others. We will use these DTOs to pass data to the StatisticService (which we will create in a moment) and the StatCommand. Let's write a StatisticService:

package com.github.codegymcommunity.jrtb.service;

import com.github.codegymcommunity.jrtb.dto.StatisticDTO;

/**
* Service for getting bot statistics.
*/
public interface StatisticsService {
   StatisticDTO countBotStatistic();
}

And its implementation:

package com.github.codegymcommunity.jrtb.service;

import com.github.codegymcommunity.jrtb.dto.GroupStatDTO;
import com.github.codegymcommunity.jrtb.dto.StatisticDTO;
import com.github.codegymcommunity.jrtb.repository.entity.TelegramUser;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.stream.Collectors;

import static org.springframework.util.CollectionUtils.isEmpty;

@Service
public class StatisticsServiceImpl implements StatisticsService {

   private final GroupSubService groupSubService;
   private final TelegramUserService telegramUserService;

   public StatisticsServiceImpl(GroupSubService groupSubService, TelegramUserService telegramUserService) {
       this.groupSubService = groupSubService;
       this.telegramUserService = telegramUserService;
   }

   @Override
   public StatisticDTO countBotStatistic() {
       List<GroupStatDTO> groupStatDTOS = groupSubService.findAll().stream()
               .filter(it -> !isEmpty(it.getUsers()))
               .map(groupSub -> new GroupStatDTO(groupSub.getId(), groupSub.getTitle(), groupSub.getUsers().size()))
               .collect(Collectors.toList());
       List<TelegramUser> allInActiveUsers = telegramUserService.findAllInActiveUsers();
       List<TelegramUser> allActiveUsers = telegramUserService.findAllActiveUsers();

       double groupsPerUser = getGroupsPerUser(allActiveUsers);
       return new StatisticDTO(allActiveUsers.size(), allInActiveUsers.size(), groupStatDTOS, groupsPerUser);
   }

   private double getGroupsPerUser(List<TelegramUser> allActiveUsers) {
       return (double) allActiveUsers.stream().mapToInt(it -> it.getGroupSubs().size()).sum() / allActiveUsers.size();
   }
}

Please note that following SOLID at the service level, we use only the services of other entities (GroupSubService, TelgeramUserService), and not their repositories. At first glance, this may look redundant, but it is not. This way we avoid problems with object dependencies with each other. TelegramUserService didn't have findAllInactiveUsers method , so let's create it in TelegramUserService:

/**
* Retrieve all inactive {@link TelegramUser}
*
* @return the collection of the inactive {@link TelegramUser} objects.
*/
List<TelegramUser> findAllInActiveUsers();

In TelegramUserServiceImple:

@Override
public List<TelegramUser> findAllInActiveUsers() {
   return telegramUserRepository.findAllByActiveFalse();
}

The repository does not have such a method, so let's add it to the TelegramUserRepository:

List<TelegramUser> findAllByActiveFalse();

This is Spring Data, so we don't need to implement this method. The service was written, it's time to make a test for this case. We create StatisticServiceTest. Here we mock data from other services and check it, and based on the mocked data, we collect the correct GroupStatDTO object:

package com.github.codegymcommunity.jrtb.service;

import com.github.codegymcommunity.jrtb.dto.GroupStatDTO;
import com.github.codegymcommunity.jrtb.dto.StatisticDTO;
import com.github.codegymcommunity.jrtb.repository.entity.GroupSub;
import com.github.codegymcommunity.jrtb.repository.entity.TelegramUser;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;

import static java.util.Collections.singletonList;

@DisplayName("Unit-level testing for StatisticsService")
class StatisticsServiceTest {

   private GroupSubService groupSubService;
   private TelegramUserService telegramUserService;

   private StatisticsService statisticsService;

   @BeforeEach
   public void init() {
       groupSubService = Mockito.mock(GroupSubService.class);
       telegramUserService = Mockito.mock(TelegramUserService.class);
       statisticsService = new StatisticsServiceImpl(groupSubService, telegramUserService);
   }

   @Test
   public void shouldProperlySendStatDTO() {
       //given
       Mockito.when(telegramUserService.findAllInActiveUsers()).thenReturn(singletonList(new TelegramUser()));
       TelegramUser activeUser = new TelegramUser();
       activeUser.setGroupSubs(singletonList(new GroupSub()));
       Mockito.when(telegramUserService.findAllActiveUsers()).thenReturn(singletonList(activeUser));
       GroupSub groupSub = new GroupSub();
       groupSub.setTitle("group");
       groupSub.setId(1);
       groupSub.setUsers(singletonList(new TelegramUser()));
       Mockito.when(groupSubService.findAll()).thenReturn(singletonList(groupSub));

       //when
       StatisticDTO statisticDTO = statisticsService.countBotStatistic();

       //then
       Assertions.assertNotNull(statisticDTO);
       Assertions.assertEquals(1, statisticDTO.getActiveUserCount());
       Assertions.assertEquals(1, statisticDTO.getInactiveUserCount());
       Assertions.assertEquals(1.0, statisticDTO.getAverageGroupCountByUser());
       Assertions.assertEquals(singletonList(new GroupStatDTO(groupSub.getId(), groupSub.getTitle(), groupSub.getUsers().size())),
               statisticDTO.getGroupStatDTOs());
   }

}

Next, we need to update our StatCommand:

package com.github.codegymcommunity.jrtb.command;

import com.github.codegymcommunity.jrtb.command.annotation.AdminCommand;
import com.github.codegymcommunity.jrtb.dto.StatisticDTO;
import com.github.codegymcommunity.jrtb.service.SendBotMessageService;
import com.github.codegymcommunity.jrtb.service.StatisticsService;
import com.github.codegymcommunity.jrtb.service.TelegramUserService;
import org.springframework.beans.factory.annotation.Autowired;
import org.telegram.telegrambots.meta.api.objects.Update;

import java.util.stream.Collectors;

/**
* Statistics {@link Command}.
*/
@AdminCommand
public class StatCommand implements Command {

   private final StatisticsService statisticsService;
   private final SendBotMessageService sendBotMessageService;

   public final static String STAT_MESSAGE = "✨<b>Подготовил статистику</b>✨\n" +
           "- Количество активных пользователей: %s\n" +
           "- Количество неактивных пользователей: %s\n" +
           "- Среднее количество групп на одного пользователя: %s\n\n" +
           "<b>Информация по активным группам</b>:\n" +
           "%s";

   @Autowired
   public StatCommand(SendBotMessageService sendBotMessageService, StatisticsService statisticsService) {
       this.sendBotMessageService = sendBotMessageService;
       this.statisticsService = statisticsService;
   }

   @Override
   public void execute(Update update) {
       StatisticDTO statisticDTO = statisticsService.countBotStatistic();

       String collectedGroups = statisticDTO.getGroupStatDTOs().stream()
               .map(it -> String.format("%s (id = %s) - %s подписчиков", it.getTitle(), it.getId(), it.getActiveUserCount()))
               .collect(Collectors.joining("\n"));

       sendBotMessageService.sendMessage(update.getMessage().getChatId().toString(), String.format(STAT_MESSAGE,
                       statisticDTO.getActiveUserCount(),
                       statisticDTO.getInactiveUserCount(),
                       statisticDTO.getAverageGroupCountByUser(),
                       collectedGroups));
   }
}

Here we are simply compiling all the information from the GroupStatDTO into a message for the admin. Since now we have StatCommand - not just a command, we need to write a separate test for it. Rewrite StatCommandTest:

package com.github.codegymcommunity.jrtb.command;

import com.github.codegymcommunity.jrtb.dto.GroupStatDTO;
import com.github.codegymcommunity.jrtb.dto.StatisticDTO;
import com.github.codegymcommunity.jrtb.service.SendBotMessageService;
import com.github.codegymcommunity.jrtb.service.StatisticsService;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;

import java.util.Collections;

import static com.github.codegymcommunity.jrtb.command.AbstractCommandTest.prepareUpdate;
import static com.github.codegymcommunity.jrtb.command.StatCommand.STAT_MESSAGE;
import static java.lang.String.format;

@DisplayName("Unit-level testing for StatCommand")
public class StatCommandTest {

   private SendBotMessageService sendBotMessageService;
   private StatisticsService statisticsService;
   private Command statCommand;

   @BeforeEach
   public void init() {
       sendBotMessageService = Mockito.mock(SendBotMessageService.class);
       statisticsService = Mockito.mock(StatisticsService.class);
       statCommand = new StatCommand(sendBotMessageService, statisticsService);
   }

   @Test
   public void shouldProperlySendMessage() {
       //given
       Long chatId = 1234567L;
       GroupStatDTO groupDto = new GroupStatDTO(1, "group", 1);
       StatisticDTO statisticDTO = new StatisticDTO(1, 1, Collections.singletonList(groupDto), 2.5);
       Mockito.when(statisticsService.countBotStatistic())
               .thenReturn(statisticDTO);

       //when
       statCommand.execute(prepareUpdate(chatId, CommandName.STAT.getCommandName()));

       //then
       Mockito.verify(sendBotMessageService).sendMessage(chatId.toString(), format(STAT_MESSAGE,
               statisticDTO.getActiveUserCount(),
               statisticDTO.getInactiveUserCount(),
               statisticDTO.getAverageGroupCountByUser(),
               format("%s (id = %s) - %s подписчиков", groupDto.getTitle(), groupDto.getId(), groupDto.getActiveUserCount())));
   }

}

Here we have checked that exactly the message that we expect is being transmitted. Of course, the CommandContainer and CodeGymTelegramBot will need to be updated so that CommandStat now passes a StatisticCommand. I'll leave it up to you. But what will it look like? We run it on a test bot and write /stat, we get: Of course, this is now visible only to admins. Now it will be clearer to them what is happening with the bot.

In conclusion

Put the new SNAPSHOT version in pom.xml for the last time:

<version>0.8.0-SNAPSHOT</version>

Since the version has been updated, it means that RELEASE_NOTES needs to be updated as well: It would seem, why fill it all out? Why write a description, who will read it? I'll tell you that just writing code is only a third (or even a quarter) of the job. Because someone should then read, expand, support this code. And documentation makes it possible to do it faster and better. All code changes can be found in this pull request . What do we have left? Only the last article with minor refactorings along with a retrospective. Let's play around before the release and analyze what we have come to in these 8 months.

Thoughts on the future of the bot

While I was preparing dinner, I was thinking about the future of the telegram bot, what else was left, what I wanted to do. And I realized that I didn’t touch on the topic of saving the state of the database (backup) of the data at all, so that I could restore it in case of something. I think that's how I would like to see it? So, to automate this process as much as possible. And against the background of these thoughts, I came to the following conclusion: I want database backups to be created from time to time and stored somewhere without my participation. But where to store? Definitely, this should be done outside the docker where the base is deployed. On the main server where the docker with the application is deployed, I also don’t really want to, because something can happen to the server and that’s all, by the way. And as a result, I came up with an idea. I must say right away that I don’t know if it can be implemented or not, but I like it the most. The idea itself:

1. Backups every day (week, month or other period of time) will be backed up to a special chat in a telegram, which admins will have access to, for example. Such a distributed data warehouse))
2. Add an admin snapshot command for admin needs.
3. Add a SUPER ADMIN command that would be able to restore the database using the provided file.

Thus, database data management becomes easier. Plus, the work of the bot will become more independent of the server on which it is located, and at any time it can be deployed with a single command on any other server. But then thoughts come to mind that this is not very safe, because access to the data is quite simple. Therefore, it would be more logical to put access to only one super admin to this data. as usual, like - subscribe - bell put a star to our project , write comments and rate the article! For the penultimate time in this series of articles, I say see you soon!

List of all materials in the series at the beginning of this article.