- Client Studies OJS3 SPAST : How we managed to optimize OJS Front-end for almost 300% load speed
- 1.The Problem
- 2.Fixing the Problem
- 2.Upgrade the OJS version
- 3.Optimize query with Eager Loading and Fix N+1 query problem
- 4. Implement infinite scroll
Client Studies OJS3 SPAST : How we managed to optimize OJS Front-end for almost 300% load speed
To get the OJTRocket plugin you can visit the bottom section bellow of this post.
One of our services is to provide quality OJS, OMP, DSpace and other platform managed servers. As many of our customers get value from what we have done to their OJS, they are convinced to use our managed server.
In this case one of our customer Spast foundation has a problem in their OJS platform although they have used the other server provider but they cannot fix their problem. Previously they hosted their platform in the AWS server, but this still never fixed their problem. Because their OJS is used for the conference system, there are so many submissions that are placed on their journal homepage. OJS is forced to show all the published submissions to show the author that their conference submission is accepted by this periodic conference event held by Spast Foundation.
They came to us and consulted about this problem that their site takes sooo long on finishing the front page loading. After we tested we found that their site will load in our browser in 20 to 30 minutes and this only happens in the frontpage of their journal which is loaded with many published submissions (for about 2000 articles). This problem is very crucial for both the organizer of the event and the author. The author never knew if their submission was included in the conference or not because they were never able to load the front page.
Luckily at openjournaltheme we have diversified so we can solve problems in a variety of approaches. we have a diversified team of specialists, server specialists, coding specialists and OJS specialists. After doing our research, this article will explain our findings.
About the SPAST foundation
SPAST is a platform which was created by like-minded academicians, Scientists, Engineers and Entrepreneurs. The long-term ambition of SPAST is to empower the sustainable development goals (SDG) of the United Nations through cultivation of good research practices.
SPAST Abstract is an online open access proceedings platform for publishing conferences, workshops, symposiums, to carry out these publications they use the OJS system with version 3.2.xx, because the OJS system is sufficient to carry out the submission process and conference publications with the appropriate flow. Procedure.
Why do they use OJS3 not OCS2 by PKP for the conference system publishing system?
Well, the ocs 2 system from PKP is very capable of running a conference system publishing system, but the system from OJS 3 is more capable in terms of the submission process workflow then security bugs in the system itself which have been minimized in OJS 3 and in terms of the appearance of the submission which is more easy to understand the flow of the submission process.
After we finished migrating OCS data on our server, the main page of the SPAST site can be accessed but it takes 30 seconds – 1 minute to access that page. But note that this is possible because we are the only visitor that accesses the site.
This happens because in their OCS System there are more than 2000 articles with published status in their last 1 issue, so that when every visitor accesses the main site, the OCS system will load thousands of articles to be displayed in the current archive on the main page of the site. SPAST, this is what causes the site to be very difficult to access. At first we thought adding sources to the VPS could fix the problem, but our initial analysis wasn’t quite right. After we upgraded the VPS Specification 2 CPUs and 2GB RAM to 6 CPUs and 16GB RAM, the site was still slow to access.
How we managed to Optimize OJS Frontend load speed more than 90% ?
Spast staff complained that the home page after we did the migration of his OJS, it still very slow. It took about half a minute to load the page. if the traffic is heavy, then the site cannot be accessed at all.
Technically ojs query data articles without limiting the number that can be retrieved. For example, if there are 2000 articles in the same issue, then those 2000 articles will be displayed by ojs. This is actually caused by OJS architecture to retrieve the article list or in other words this is not caused by the theme or plugin.
On the retrieval of a submission, OJS loads all the data of the submission table, this causes that on a single query it will also populate with unneeded data that will be shown to the visitor.
Identifying Performance Bottlenecks
First, we think this is a bottleneck problem from the server.
However, after we upgraded the server from 2 CPUs and 2GB RAM to 6 CPUs and 16GB RAM, the bottlenecks persisted.
It was concluded that the cause of this problem came from OJS itself.
We dug into the ojs source code to find out where the problem was coming from. It turned out that there was code that caused the N+1 query problem (Click here for the details about the N+1 query problem).
Then We did the logging in mysql, and we found a lot of N+1 queries.
The above query was performed 4818 times, (There are 2409 submissions in 1 issue). This also happens to several other queries related to existing submissions.
OJS calls the SubmissionServiceClass to get a Submission List with some arguments as shown below.
File : /ojs/pages/issue/IssueHandler.inc.php
Then the SubmissionServiceClass performs a query into the database.
File : /ojs/lib/pkp/classes/services/PKPSubmissionService.inc.php
After getting the required data, using the DAOResultFactory class, each data submission will call the `_fromRow` function of the SubmissionDAO class.
This is where the N+1 query problem occurs, where every data submission that will query the database to get data publications related to the submission occurs, causing repeated queries to the database, and therefore causing overhead in MySQL.
File : /ojs/lib/pkp/classes/submission/PKPSubmissionDAO.inc.php
Detail About The Problem
The problem is within the query itself. The query inefficiently takes repeated data with multiple query. This is known as the N+1 query issue.
This explained very well in this article here :
The N+1 query is a query that executes an additional query to the database server that actually fetches the data that can be obtained by executing the first query.
A.Get submissions by contextId, IssueId, and Published Status
This query is only executed once by PHP. For example, here the query returns the result with 2000 submissions.
B.Get publication by submission_id
Each submission will get publication data by using this query. So, if there are 2000 submissions, this query will run 2000 times
C.Get locale from submissions table by submission_id
Each publication will get a data locale using submission_id as a parameter. So, if there are 2000 publications, This query will run 2000 times.
D.Get authors by publication_id
Each publication will get data from authors using publication_id as parameter, So, if there are 2000 publications. This query will run 2000 times.
E.Get publication_galleys by publication_id
Each publication will get data publication_galleys using publication_id as a parameter, So, if there are 2000 publications. This query will run 2000 times.
Total, the number of queries performed every time there is a request, is 8001 queries
2.Fixing the Problem
After our team did some code analysis, we found out that this is caused by the inefficient query that resided on the OJS source code. The query creates duplicate and unnecessary code that retrieves double records from many tables.
Our first thought is modifying how OJS loads the article on the first journal page that shows a list of articles. Rather than load all the articles which are about 2000 published submissions, it needs to load with limitations. One the solution for this is using the pagination feature :
But after a team discussion this can be improved, because after we think of an author role perspective that sends submission to that conference system, it will be good if the author article will be shown without needing to click and find too many pages of it. So our approach is on the user experience level (UX). Yes, we need another solution for this.
After brainstorming with our team, we agreed that we will implement an infinite scroll on the journal homepage. By implementing this solution, OJS does not need to load all the articles but it will fetch a list of articles with limited articles but always doing the fetching as the user scrolls down.
Thanks to powerful Hooks utilities by PKP Team, we don’t need to modify OJS source code, instead we create a plugin to fix this problem. Using hooks utilities we were replacing some of the default handlers using our own custom handler, with this instead using the default handler to load frontend pages, ojs will load some frontend pages using our custom handler. After that we have the control of how to implement the logic to prepare the data for the visitor. Using an eager loading method to fix N+1 query problems makes the query much more effective than before. Then we cache the data to quickly reload the data after the next visitor is coming.
Microcaching is a technique that utilizes the caching feature in the Nginx server by keeping the page in the short period of time. By keeping the page as a static file, it will decrease the proxy process or data query to another app such as running logical instruction to PHP or requesting multiple data / query to MySql server.
By using this technique the temporary static file can be used by other visitors so it also decreases the load time for another user. To make sure that the content is fresh, we set up the caching period for only a few seconds.
We have integrated the OJS set up with Nginx Microcache so the client will benefit faster and thousands of visitors without overloading the server stack.
2.Upgrade the OJS version
Currently our client uses the outdated version of OJS which is 3.2.x version. On using this version the journal homepage never finishes loading the article. Seems like there is more overhead code in the OJS source code on this version. First of all we need to upgrade this OJS to the latest version.
And yes it is optimized with 100% speed improvement on our local machine. However, since this is just in the local environment. After simulating it on the production server, this conference page still never gets loaded, just doing the infinite loading and overload on the hardware (Mysql) in the background.
Upgrading the client’s OJS is just a simulation to know if there is any speed improvement from the new OJS version. However, as we are aware that the latest OJS 3.3.x has a critical bug for announcement sending, this upgrade step is not an option for us. We have communicated this with our client and the decision is to keep the OJS with their current version.
3.Optimize query with Eager Loading and Fix N+1 query problem
Eager loading is merely fetching records preemptively. For example, when the submission records are requested, you can aggregate their IDs. Then, you can query for only the publications used by those submissions beforehand.
For example :
This query returns the submission IDs 1,2,3,4,5 instead of querying each submission id to get related publications.
You can save queries on publications by running just on query statement
Using this method we optimize the query and fix the N+1 query problem.
4. Implement infinite scroll
Loading 2000 or more data in one request is an inefficient way to do so. We think of paginating the page so not all data is loaded in a single request. But after that we realize that it is not the best way to do it. Because it needs user interaction to navigate between pages. Instead we came up fwith an infinite scroll method, so every time scrolling to the bottom of the page, it will automatically load more data.
Since migrating servers and optimizing the OCS system on openjournaltheme using OJTRocket pugin, Openjournaltheme has made spast websites easier and faster to access, with that we believe it will have a very positive impact on their site visitors, with easy access sites and increase the number of authors on their sites.
Using Openjournaltheme server and OJTRocket plugin, the SPAST Foundation achieved a transparent data migration to Openjournalteheme without impact to system submission operations. Openjournaltheme helps maintain the highest access of their system by working to melakukan implementasi lazyload pada halaman utama mereka.
After optimized :
After optimized + cache :
After optimized + implement infinite scrolling :
The solution from our research results, we manifest in the form of a plugin that we call OJTRocket Plugin which optimizes the query process for the list of articles, you can get this plugin by visiting the following page:
We believe that the plugin we have created can help the OJS community who are experiencing the same problem.
Quick steps for using this plugin
- Download OJTRocket Plugin
- Upload the plugin from the admin dashboard Website Settings -> Plugins -> Upload a New Plugin or unpack the archive into the plugins/generic/ folder.
- Activate the plugin from the dashboard
- Clear Cache Template
- This Plugin supports OJS 3.2 and OJS 3.3
What are the benefits of this optimization :
1. Google will reward sites that have high access speeds, this will have an impact on the SEO of your journal site
2. Authors can find their articles more easily because the process of loading articles on the archive becomes faster
3. visitors will be more comfortable looking for an article in your journal because the process of loading articles on the archive becomes faster
GET THIS PLUGIN
Subscribe to get this Plugin and our product release!
Our services that we provide are always oriented to the needs of our users, in this case we take various approaches both in terms of server optimization, security, and in coding improvement in OJS as we have explained. This effort is our contribution so that we can help each of our clients’ specific needs in an effort to disseminate knowledge more broadly.
Honestly we just know that there are publishers that creatively use OJS as a conference system platform so we think that this can be a new idea for any publisher that needs to find a free platform for their conference system.
As a form of our contribution to the community, we also provide a plugin that we developed to overcome the N+1 issue in OJS which we call OJTRocket. We provide this plugin for free. This plugin optimizes OJS queries, especially on the front page of journals that load a lot of articles.
Put your email in this form and our system will send this plugin without any cost.
We hope that this plugin can help you as a publisher, both for those of you who use OJS with a lot of submissions and those who use the OJS platform as a replacement platform for the OCS / conference system.
Contact us if you need special needs related to your OJS.
Thank you .