How we managed to Optimize OJS Frontend for almost 300% load speed 😲

Client Studies OJS3 SPAST: How we managed to optimize OJS Front-end for almost 300% load speed

To get the OJTRocket plugin you can visit the bottom section below this post.

One of our services is to provide quality OJS, OMP, DSpace, and other platform-managed servers. As many of our customers get value from what we have done to their OJS, they are convinced to use our managed services.

In this case, one of our customers Spast foundation has a problem with their OJS platform although they have used the other service provider they cannot fix their problem. Previously they hosted their platform in the AWS server, but this still never fixed their problem. Because their OJS is used for the conference system, there are so many submissions that are placed on their journal homepage. OJS is forced to show all the published submissions to show the author that their conference submission is accepted by this periodic conference event held by Spast Foundation.

They came to us and consulted about the problem that their site takes so long on finishing the front page loading. After we tested we found that their site will load in our browser in 20 to 30 minutes and this only happens on the front page of their journal which is loaded with many published submissions (for about 2000 articles).  This problem is very crucial for both the organizer of the event and the author. The author never knew if their submission was included in the conference or not because they were never able to load the front page.

Luckily at openjournaltheme, we have diversified so we can solve problems in a variety of approaches. we have a diversified team of specialists, server specialists, coding specialists, and OJS specialists. After doing our research, this article will explain our findings.

About the SPAST foundation

SPAST is a platform that was created by like-minded academicians, Scientists, Engineers, and Entrepreneurs. The long-term ambition of SPAST is to empower the sustainable development goals (SDG) of the United Nations through the cultivation of good research practices.

SPAST Abstract is an online open-access proceedings platform for publishing conferences, workshops, and symposiums, to carry out these publications they use the OJS system with version 3.2.xx because the OJS system is sufficient to carry out the submission process and conference publications with the appropriate flow. Procedure.

Why do they use OJS3 not OCS2 by PKP for the conference system publishing system?

Well, the OCS 2 system from PKP is very capable of running a conference system publishing system, but the system from OJS 3 is more capable in terms of the submission process workflow than security bugs in the system itself which have been minimized in OJS 3 and in terms of the appearance of the submission which is easier to understand the flow of the submission process.

Problem: very slow site load

After we finished migrating the OJS data on our server, the main page of the SPAST site can be accessed but it takes 30 seconds – 1 minute to access that page. But note that this is possible because we are the only visitor that accesses the site.

Note that the client uses the OJS for managing their conference system.

This happens because in their OJS System there are more than 2000 articles with published status in their last 1 issue so when every visitor accesses the main site, the OCS system will load thousands of articles to be displayed in the current archive on the main page of the site. SPAST is what causes the site to be very difficult to access. At first, we thought adding sources to the VPS could fix the problem, but our initial analysis wasn’t quite right. After we upgraded the VPS Specification 2 CPUs and 2GB RAM to 6 CPUs and 16GB RAM, the site was still slow to access.

How do we manage to Optimize OJS Frontend load speed by more than 90%?

Spast staff complained that the home page after we did the migration of his OJS, it still very slow. It took about half a minute to load the page. if the traffic is heavy, then the site cannot be accessed at all.

Technically ojs query data articles without limiting the number that can be retrieved. For example, if there are 2000 articles in the same issue, then those 2000 articles will be displayed by ojs. This is actually caused by OJS architecture to retrieve the article list in other words, this is not caused by the theme or plugin.

On the retrieval of a submission, OJS loads all the data of the submission table, this causes that on a single query it will also populate with unneeded data that will be shown to the visitor.

Identifying Performance Bottlenecks

First, we think this is a bottleneck problem from the server.

However, after we upgraded the server from 2 CPUs and 2GB RAM to 6 CPUs and 16GB RAM, the bottlenecks persisted.

It was concluded that the cause of this problem came from OJS itself.

We dug into the ojs source code to find out where the problem was coming from. It turned out that there was code that caused the N+1 query problem (Click here for the details about the N+1 query problem).

Then We did the logging in mysql, and we found a lot of N+1 queries.

The above query was performed 4818 times, (There are 2409 submissions in 1 issue). This also happens to several other queries related to existing submissions.

Analyzing issues

OJS calls the SubmissionServiceClass to get a Submission List with some arguments as shown below.

File : /ojs/pages/issue/IssueHandler.inc.php 

Then the SubmissionServiceClass performs a query into the database.

File : /ojs/lib/pkp/classes/services/PKPSubmissionService.inc.php

After getting the required data, using the DAOResultFactory class, each data submission will call the `_fromRow` function of the SubmissionDAO class.

This is where the N+1 query problem occurs, where every data submission that will query the database to get data publications related to the submission occurs, causing repeated queries to the database, and therefore causing overhead in MySQL.

File : /ojs/lib/pkp/classes/submission/PKPSubmissionDAO.inc.php

Detail About The Problem

The problem is within the query itself. The query inefficiently takes repeated data with multiple queries. This is known as the N+1 query issue.

This is explained very well in this article here :

The N+1 query is a query that executes an additional query to the database server that actually fetches the data that can be obtained by executing the first query.

A.Get submissions by contextId, IssueId, and Published Status

This query is only executed once by PHP. For example, here the query returns the result with 2000 submissions.

B.Get publication by submission_id

Each submission will get publication data by using this query. So, if there are 2000 submissions, this query will run 2000 times

C.Get locale from submissions table by submission_id

Each publication will get a data locale using submission_id as a parameter. So, if there are 2000 publications, This query will run 2000 times.

D.Get authors by publication_id

Each publication will get data from authors using publication_id as parameter,  So, if there are 2000 publications. This query will run 2000 times.

E.Get publication_galleys by publication_id

Each publication will get data publication_galleys using publication_id as a parameter,  So, if there are 2000 publications. This query will run 2000 times.

Total, the number of queries performed every time there is a request, is 8001 queries

Fixing the Problem

After our team did some code analysis, we found out that this is caused by the inefficient query that resided on the OJS source code. The query creates duplicate and unnecessary code that retrieves double records from many tables. 

Our first thought is to modify how OJS loads the article on the first journal page that shows a list of articles. Rather than load all the articles which are about 2000 published submissions, it needs to load with limitations. One the solution for this is using the pagination feature :

 But after a team discussion this can be improved because after we think of an author role perspective that sends submission to that conference system, it will be good if the author’s article will be shown without needing to click and find too many pages of it. So our approach is on the user experience level (UX). Yes, we need another solution for this.

 After brainstorming with our team, we agreed that we will implement an infinite scroll on the journal homepage. By implementing this solution, OJS does not need to load all the articles but it will fetch a list of articles with limited articles but always do the fetching as the user scrolls down.

 Thanks to powerful Hooks utilities by PKP Team, we don’t need to modify the OJS source code, instead, we create a plugin to fix this problem. Using hooks utilities we were replacing some of the default handlers using our own custom handler, with this instead of using the default handler to load frontend pages, ojs will load some frontend pages using our custom handler. After that, we have control of how to implement the logic to prepare the data for the visitor. Using an eager loading method to fix N+1 query problems makes the query much more effective than before. Then we cache the data to quickly reload the data after the next visitor is coming.

1. NGINX Micro cache

Microcaching is a technique that utilizes the caching feature in the Nginx server by keeping the page for a short period of time. By keeping the page as a static file, it will decrease the proxy process or data query to another app such as running logical instructions to PHP or requesting multiple data/queries to the MySql server. 

By using this technique the temporary static file can be used by other visitors so it also decreases the load time for another user. To make sure that the content is fresh, we set up the caching period for only a few seconds.

We have integrated the OJS set up with Nginx Micro cache so the client will benefit faster and thousands of visitors without overloading the server stack.

2. Upgrade the OJS version

Currently, our client uses the outdated version of OJS which is the 3.2.x version. On using this version the journal homepage never finishes loading the article. Seems like there is more overhead code in the OJS source code on this version. First of all, we need to upgrade this OJS to the latest version. 

And yes it is optimized with 100% speed improvement on our local machine. However, since this is just in the local environment. After simulating it on the production server, this conference page still never gets loaded, just doing the infinite loading and overload on the hardware (MySql) in the background.

Upgrading the client’s OJS is just a simulation to know if there is any speed improvement from the new OJS version. However, as we are aware that the latest OJS 3.3.x has a critical bug for announcement sending, this upgrade step is not an option for us. We have communicated this with our client and the decision is to keep the OJS with its current version.

3. Optimize query with Eager Loading and Fix N+1 query problem

Eager loading is merely fetching records preemptively. For example, when the submission records are requested, you can aggregate their IDs. Then, you can query for only the publications used by those submissions beforehand.

For example :

This query returns the submission IDs 1,2,3,4,5 instead of querying each submission id to get related publications.

You can save queries on publications by running just on the query statement

Using this method we optimize the query and fix the N+1 query problem.

4. Implement the infinite scroll

Loading 2000 or more data in one request is an inefficient way to do so. We think of paginating the page so not all data is loaded in a single request. But after that, we realize that it is not the best way to do it. Because it needs user interaction to navigate between pages. Instead, we came up with an infinite scroll method, so every time scrolling to the bottom of the page, it will automatically load more data.

5. Result

Since migrating servers and optimizing the OCS system on openjournaltheme using the OJTRocket plugin, Openjournaltheme has made Spast websites easier and faster to access, with that we believe it will have a very positive impact on their site visitors, with easy access sites and increase the number of authors on their sites.

Using the Openjournaltheme server and the OJTRocket plugin, the SPAST Foundation achieved a transparent data migration to Openjournalteheme without impact on system submission operations. Openjournaltheme helps maintain the highest access of their system by working to implement the lazyload to their article list page.

Before :

After optimized :

After optimized + cache :

After optimized + implement infinite scrolling :

Conclusion :

The solution from our research results, we manifest in the form of a plugin that we call OJTRocket Plugin which optimizes the query process for the list of articles, you can get this plugin by visiting the following page:

We believe that the plugin we have created can help the OJS community who are experiencing the same problem.

Quick steps for using this plugin

1. Download OJTPlus Plugin here
2.  Upload the plugin from the admin dashboard Website Settings -> Plugins -> Upload a New Plugin or unpack the archive into the plugins/generic/ folder.
   

   

3. Activate the plugin from the dashboard
4. Clear Cache Template
   

   

Why the load speed is a matter for your journal?

1. Google will reward sites that have high access speeds, this will have an impact on the SEO of your journal site
2. Authors can find their articles more easily because the process of loading articles on the archive becomes faster
3. Visitors will be more comfortable looking for an article because the load is faster
4. Any indexing may exclude the journal from their indexing since the site cannot be reached because of the performance issue.

Get the plugin

Subscribe to get this Plugin and our product release!

The services that we provide are always oriented to the needs of our users, in this case, we take various approaches both in terms of server optimization, security, and coding improvement in OJS as we have explained. This effort is our contribution so that we can help each of our client’s specific needs in an effort to disseminate knowledge more broadly.

Honestly, we just know that there are publishers that creatively use OJS as a conference system platform so we think that this can be a new idea for any publisher that needs to find a free platform for their conference system.

As a form of our contribution to the community, we also provide a plugin that we developed to overcome the N+1 issue in OJS which we call OJTRocket. We provide this plugin for free. This plugin optimizes OJS queries, especially on the front page of journals that load a lot of articles.

Put your email in the above form and our system will send this plugin without any cost.

We hope that this plugin can help you as a publisher, both for those of you who use OJS with a lot of submissions and those who use the OJS platform as a replacement platform for the OCS / conference system.

Contact us if you need special needs related to your OJS.

Plugin Compatibility : 

• This Plugin supports OJS version 3.2 and OJS version 3.3

Thank you.

OJT Hosting By Openjournaltheme

If you are feel that handling such kind of technical issue is not necessary and need a dedicated team to handle all the technical things.

We have developed a hosting that is specifically created for OJS/OMP platform that is implement caching and secure system for you to be able to manage the platform easier without need detailed technical debt/maintenance, read the detail here.