Server performance guide

§1. Asking for help

Still needing help in configuring or finding out? Here’s a few guidelines and tips before you ask for help from devs/moderators/forum posts/discord users:

1. Always have your logs. Most of the logs will be located in ./logs folder, and the one you generally will want at hand is the “debug” or otherwise called “debug.log”. You could also download the “latest” or “latest.log”.

   1. Once you have them open or downloaded, please upload them to Pastebin or Github (the latter needs a Github account). This will make it easier to share.

2. If you are having specific issue with performance (aka lag), please make sure you read the ENTIRE guide.

   1. If your server is lagging after having followed ALL the directions, please run a Spark and Timings (you know what that is, you read the entire guide… right?). Post those links in your initial support request message.

3. Lastly, here is the most important takeaway from this section: where should you ask for help? This entirely depends on what is the issue, so here is a general ‘resource’ link list based on need. Please remember to add the relevant links discussed prior.

   1. Pixelmon Discord: use this discord when you need help with Pixelmon-related issues, or if you want to setup/need help setting up your server with Pixelmon. If you are having isssues with performance AND you run Pixelmon, you can also ask for help there. Please remember to use the “#server-support” channel.

   2. Sponge Discord: use this for fast-help related to Sponge issues. This includes, but is not limited to, configuring or developing for Sponge, performance issues, console spam (which normally SAYS if it involves Sponge). If you use other mods other than Pixelmon, I would also recommend this as a ‘starting’ point from where you can see if other Sponge+’Insert mod you use’ users are having the same issue(s) as you.

      1. If you don’t get a reply there (which is possible, there is no support channel so you have to be patient), you can always post to their forums instead.

   3. Nucleus Discord: if you are having issues with NUCLEUS RELATED FEATURES*, join there and use one of the “quiet-support” channels for additional clarity. Please make sure to state what version of Nucleus you are running.

      1. *Nucleus related features imply it’s a NUCLEUS addition, not a Sponge/Minecraft/Something feature.

      2. Make sure to read the Nucleus docs before posting, 99% of the time the answer is already there.

   4. Luckperms Discord: if you are having issues with Luckperms, join there and use one of the “general” channels. Please make sure to state what version of Luckperms and Sponge you are running.

      1. Make sure to read the Luckperms wiki before posting, literally everything is explained there.

   §2. Pre-generating your Map

   Note, this CANNOT be avoided. This will sharply reduce the lag caused by your players exploring the map.

   1. Download and install Nucleus, the Sponge-Essential plugin.

   2. In your server.properties (located in the root, or main server folder), set the “max-tick-time” to -1. This will prevent your server from timing out during this process. This setting is used to tweak the Server Watchdog, a Minecraft ‘feature’ that forcefully crashes your server if a single tick takes too long. While pregenerating, it is very likely to happen, but this is normal and expected. Disabling it removes the possible annoying crashes.

   3. Ensure that your server is empty, or is whitelisted. Players online will lag a lot, so best leave it empty.

   4. Use /world border set [world] [center-x] [center-z] [diameter-in-blocks]. The bigger the border, the bigger the map, the longer the pre-generating process will take. I recommend anything between 15000 to 25000. Once you have decided on your border's size, add 2000 blocks more to its count. If you want 15000, then 17000 will be your border size for pregenerating.

      1. Size choices vary depending on A) space available on your host, B) the number of players this map will be for. Below are some notes you should keep in mind:

         1. In general, anything above 30K in diameter will be a pain to manage, as worlds of that size can go upwards of 50GB after 10 months of use.

         2. I’ve personally always limited myself to 26K in diameter (before the border buffer) for 100 players max online. I’ve seen bigger and smaller, though I have never ran out of space that way. For a smaller server population (or if you are just starting out), 15K in diameter should be fine.

         3. Remember that this pregeneration process has to be replicated for every world that will be used by players. Meaning, 3 worlds of 26K each is probably not the best usage of space.

         4. Lastly, it is ALWAYS good to keep one, or two backups. Even zipped, their size will likely decrease 30% to 40%, meaning you cannot forget to account for the space they will take on your machine. Don’t forget that when you are choosing your world size.

   5. Use /world border gen [world] to start the pregenerating. You will see messages in console indicating the progress. Don't worry, it'll stop once it reaches your border, so you can basically leave your server until it is done.

      1. The -a flag is used to pregenerate in “aggressive” mode, meaning it will go faster, but it will also be straining the server a lot more. Only use it if your server host is not a potato. Example would be: /world border gen -a [world].

      2. You can also increase save time to accelerate the generation process, it will however imply you will lose more time if the server ends up crashing. Example would be: /world border gen --save 5m [world]. You can combine this with the -a flag to generate at the fastest rate possible.

   6. Expect the pre-generating process to take a long time. Make yourself a coffee, or two, or three. Wait.

   7. Once the completion message is show in console, use /world border set [world] [center-x] [center-z] [diameter-in-blocks]. The diameter here will be, for instance, your initial 15000 blocks in diameter. This will prevent players at the border from generating past it, and act as a buffer to ensure no more generation takes place.

   8. Repeat this for as many worlds as you have.

   9. Once you are done, go back to your server.properties and set the “max-tick-time” to 18000.

   §3. Proper JVM Arguments

   1. I'll let Aikar here take it away for me. He's the guy that did the timings for Sponge, and has generally been a bounty of information on Java performance in the Minecraft community. Read about it here

   2. One thing I will mention is that nowadays, most modded servers will need at least 2GB to run a server. The more players you have online, the more likely it is that your RAM will need increasing. Do NOT assign all RAM on your host, and do NOT assign above 20GB. I've seen servers run at 20 TPS with 100 players with -xmx16GB and -xms14GB, so it is definitely possible. If you need to assign more, something's wrong and it's not your RAM's fault.

   §4. Reduced View-Distance

   1. Open your server.properties file, located at the base (root) folder of your server).

   2. By default, Minecraft's view-distance is of 10. This means that around the player, a radius of 10 chunks will be loaded as they move/idle. That's a lot of chunks.

   3. The more chunks are loaded, the worse it is for your server. Decrease the value of the view-distance to 6, or 4.

   §5. Tile-Entity and Entity Activation Ranges

   Sponge's configuration file, the global.conf (located under /config/sponge) allows you to change activation ranges of both entities and tileentities. This helps the server process less tasks per tick, and should greatly improve your performance. Note: an ‘activation range’ is the radius around the tile-entity (for example, a furnace) or an entity (for example, a creeper) for which it will tick when in proximity to the player. If you set your radius for example to 1, it will mean that your furnaces will only update when at 1 block around the player; of course, that also means your players will be very annoyed when their farms don’t update because the radius is too low. Similarly, if you have a creeper with an activation range of 0, chances are it will never update, and just explode when the player is standing right on top of it.

   1. Open the global.conf.

   2. Under "entity-activation", turn on the `auto-populate` feature. Change the default range to 32.

   3. Find "tileentity-activation", and do the same - turn on the `auto-populate` argument, and set the default range to 32.

   4. Save and upload the file to your server, making sure to restart.

   5. Periodically check that file over the next couple of hours or days (depending how active your players are). The config will start populating, and more options will appear, under the name of entities (minecraft:pig) or tile-entities (pixelmon:fossil_cleaner).

   6. Now, you can lower or raise the ranges of the blocks defined in the global.conf, under "entity-activation" and "tile-entity-activation". I normally lower most to 16/8/6, but that's because I like to live on the edge. The lower the value, the less strain those items have on your server, but that doesn't mean you should lower it to almost nothing. That would cause your entities/tile-entities to malfunction. We don't want that.

   *** Note: Sponge’s config files are kept alive during runtime, meaning that the server prevents you from editing them. On shutdown, these files revert. If you are editing values and your server is running, please do /sponge -g reload to reload the configuration and save your changes. Alternatively, you can edit these files while offline. I would recommend editing offline, because Sponge configs require a restart to be effective.

   §6. Choosing your Plugins or Mods

   This comes down to the quality of what your server is running. Not all plugins, or mods, have the same impact on your performance. Some, like Pixelmon, are unavoidable costs on our networks, but it might not be the case with plugins, or mods.

   In comparing Polis to GriefPrevention, the former is heavy on the server because it logs everything into a single file, whereas the latter divides the checks into many files that are managed by Luckperms. Choosing the right plugins (and mods) for your server will be one of the ways that you can greatly hinder, or improve, your performance. If you have questions about something, ask around, see if anyone has commentary on its performance on a server.

   The golden rule is this: nothing is free. If you have a plugin with an inventory GUI, or block-logging features, or even regeneration, that stuff CAN be expensive. You have to be prepared for features added to impact your server’s performance. Remember, Minecraft is a single-core game, and no amount of additional RAM will make it go any faster on the main thread. Choosing your plugins and mods wisely will help you decide what you keep, and what is too costly to run.

   §7. Tinkering Plugin/Mod Configs and Options

   There's always new features for you to test and experiment with. The best way you can keep a good server performance is to watchout for new methods of lowering your server's strain. Of course, please ask around if you are going to edit something you have no idea about. Here are a few I've found.

   1. Lighting async values can be found in Sponge's global.conf. You can assign more threads to this, and it should help with the lighting updates within loaded chunks.

      1. If you don’t know what this implies, or what it can potentially do, please do not use this option.

   2. Real-time TPS is an experimental feature in Sponge's global.conf as well. In that file, under "modules", set `real-time` to true. It'll try to replicate as much as possible 20 TPS for the client, even when the server is lagging.

   3. Sometimes, GriefPrevention causes issues with the buildup of dropped items. Use /cf collide-block minecraft:item minecraft:any true override in your wilderness to lower their impact on the server (thanks to I9 for this!).

   4. Use CatClearLag, or a command scheduler, to queue /pokekill and dropped-item despawning. Both are notoriously fickle, so I recommend giving them the extra help needed to make sure those are recycled.

      1. CatClearLag, along with other lag-fixing plugins are known to work off of basic config fixes that you could be doing yourself, like the spawning rate in Pixelmon, or base MC mobs. I personally would not recommend using it as an ‘improving’ plugin, as it does not actually clear your lag. That said, if you need the entity clearing scheduling because, for instance, entity despawning is an issue, use it. It’s a good way to schedule clearing tasks, but don’t expect it to be magic. It won’t replace anything in this guide.

   5. World saves are a real issue on any server. By default on Sponge, they happen every 45 seconds. In your global.conf, change the tick count between player-data and world save to the SAME amount - I usually do every 5 minutes, so `6000` ticks.

      1. You could be scheduling a longer interval, it’s a matter of personal decision. Just remember to think of this interval as “the maximum amount of timed progress my players are willing to lose in the instance of a crash”. If the server crashes, the data is reverted to the last world save - so you’d want these as frequently as can be managed.

   6. In Sponge’s global.conf, enable structure saving and let it auto-populate to true. It’ll help saving mineshafts.

      1. Sponge also has a `use-real-time` tick option, as well as `user-panda-redstone` and ‘eigen-redstone’ fix option. All three are recommended.

         1. If you experience weird behaviour with redstone, please try first disabling panda and eigen fixes to see if the issue persists. If it fixes the glitches you see, please report to Sponge.

   7. Pixelmon Settings

      1. Pixelmon’s hocon config has a great way to lower issues related to saves: turn on async pokemon saving. Search for `useAsyncSaving` and set it to true.

         1. If you see "ComputerManager" or "PokeballManager" as the highest two entries in your timings, this means you did not follow the guide fully. Read please!

      2. Pixelmon Reforged's Better-Spawner is a treasure for performance. Play around with the different options, and see what works best for you. All servers are different, and all players want different things. It's up to you to find a good balance. Check this page for information on the Better-Spawner config options in the pixelmon.hocon (located under /config/pixelmon.hocon)

      §8. Keeping Track of Performance

      Performance is never a sure thing, and it will vary constantly, so you have to be prepared to recognise and ascertain issues as they arise. You can do this by monitoring your server, and keeping an eye on errors, or values out of the ordinary.

      1. Sponge Timings

         1. In your Sponge global.conf, under the "timings" section, set the `history-length` to 18000, and the interval to `2000`.

         2. This will allow more data to collect in your Sponge Timings, which you can view by using /sponge timings paste. Though this might look complicated at first, Sponge Timings are a useful tool to find out performance issues. Make sure that the right-corner header is set to "AVG" and "ALL". Intuitively, the graph should be mostly green, with no major fluctuations. Here’s an example.

         3. The data under it, sorted into a table, should not contain values out of the ordinary. If you see "doChunkMap", "chunkGenerator" or "chunkPopulate", you didn't fully pre-generate your map (see above on how to do this).

      2. Nucleus’ GC Command

         1. Nucleus’ /gc -c command is a great way to quickly assess performance in the last session. Always note the average "loaded chunk" count, which will vary depending on the number of players.

      3. Forge & Sponge’s TPS Command

         1. Both Forge and Sponge have a command to monitor TPS per world. Use /forge tps or /sponge tps to quickly check the server's current performance.

         2. Minecraft counts 20 ticks per second (TPS), so if the resulting output shows it to be below 20 TPS, run a timings and check your console for clues as to what is happening.

      4. Server Console

         1. The most primitive tool left, the server console, is only ever useful when you see a recurring error appear as your TPS decreases. Most, but not all, spamming errors will affect your performance, so make sure that you have checked for errors in the console before further investigating.

         2. If you see "moved too quickly!" messages, then your server is suffering from chunk loading. Make sure your map is pre-generated to lessen the impact of chunk loading.

      5. TickProfiler

         1. Pretty simple tool to use, it is installed as simple jar. You can find the wiki and it’s open source on MinimallyCorrect’s github. It will be a useful tool that can help when timings do not seem to give you a good idea of what exactly is causing all that TPS loss. It is used to track entity and tile-entity usage. Thanks to merjilin for this tip.

      6. WarmRoast

         1. Recommended only as a profiling tool NOT to leave online at all times, you can download and install it from its author’s Github, sk89q. This is mostly useful when you have a memory leak and timings cannot seem to decide what’s wrong.

      7. Spark

         1. This plugin for Sponge was developed by Luck, the same person that made Luckperms. A more user-friendly option to WarmRoast, this allows process profiling in-game (which WarmRoast does not). The added accessibility makes this my preferred option, as it is easy to setup and has a discord if ever you need more help. Download and read the wiki here.

            1. A simple command to use is /spark sampler --thread * on. This sampler will have to be stopped once you’ve neared 3-5 minutes of runtime DURING lag spikes.

      8. Sampler

         1. Arguably the most useful out of the bunch in my personal experience, Sampler allows server owners to set a TPS timer and catch an all-threads, spark-like memory snapshot that is registered to a file. It also automatically will generate stall-reports, useful for never-ending spikes, and effectively replacing that ‘watchdog’ crash report that is pure garbage (sorry to burst anyone’s bubble). It’s a free, public plugin, which can be obtained here. Simply drop it to your /mods folder to install.

            1. To get a sampling report, run /sampler trigger ticktime 30000. This command will therefore take a snapshot nps when the ticktime reaches 30000. If you want to be more aggressive, you can always set this lower. You will have to do this every restart (and note, as all sampling plugins, running it does impact your performance slightly, so I wouldn’t run it just for fun). That command will generate a `.nps` file in your server’s root directory.

               1. To open it, you will have to download VisualVM (free software). Once you do, you can unzip to your desktop the downloaded file, and run the exe.

               2. Once you have VisualVM open, you can open your .nps file, and start poking at the data collected.

               3. My personal trick is to go into ‘Select Threads’, uncheck everything in the dropdown list and keep ‘Server_Thread’ checked. The data will then be re-sorted automatically.

               4. Just like a Spark, you can then collapse and expand tabs to look at the possible lag culprits, like this.

            2. As mentioned previously (and if you didn’t see this, read back up), stall-reports are created automatically. They are located in the /stall-reports folder in your server’s directory, and they are simple text files, like crash-logs or debug logs.

               1. Download and start opening them (I like Notepad++ because of its ez pz layout), and check for the first sections of the stall report to see the possible sources.

               2. Though this is slightly less accurate than sampling reports, it can be useful in determining a slowdown in console, and usually will be created in batches. For better accuracy, consider using the FIRST stall-report out of a single ‘batch’, since all the subsequent stall-reports are usually slowdowns caused by the initial performance issue.

      
      

      §9. Fixing Lost Stability (Crashes, timeouts, TPS loss and other painful things)

      There is no such thing as perfect stability. Issues always arise, and when you start noticing that your performance is going downhill, act fast. Certain issues, like the ones caused after crashes, can cripple your server and your players, making it hard to recover. In the interest of not cluttering endlessly this section, only common Minecraft/Forge/Sponge issues that are not plugin related will be covered.

      1. Common Crash causes

         1. Corrupted regions or playerfiles

            1. A region file may be completely broken, to the point that it is best to delete it. In your worldname/region or worldname/playerdata folder, please check for any region files (.mca files) with a size of 0. If there are any, delete or restore from backup. These are broken files, and they will crash you if the world save tries to process them.

            2. Sometimes, chunks may not appear problematic at the first glance. Use Fenixin’s Region Fixer to run your world through the checkup process. Any issues it might find with the files will point you to regions that are broken.

         2. Corrupted or malformed level.dats

            1. Probably the scariest issue out there for most, this one has the particular ‘feature’ of completely changing your modded block IDs to other blocks. Like all Apricorn trees might become Bone blocks, and so on. To recover:

               1. Shutdown your server.

               2. From a backup, grab the “level.dat”, “level_sponge.dat”, “level.dat_old”, “level_sponge.dat_old” and the “forcedchunks.dat”. Of course, if you don’t run a Sponge server, no _sponge dat will be found.

               3. With the server STILL OFFLINE, replace all 5 files from your backup to your main server. If you have multiple worlds affected, repeat the process as needed, as these 5 files are different for each world.

               4. Start up the server. The blocks should be back to normal.

         3. Server timeouts in 1.12+: Random Patches

            1. If your server has player timeout issues, try this (server-side) mod, by TheRandomLabs. It should fix the Mojang hardcoded server timeout on login. Read more of it here, and here.