What's new?
This week brings a handful of critical fixes to the server. This includes temporarily rolling back Riposte damage so NPCs aren't one-shotting you, fixing a rare exception that could cause a zone to crash, and fixing a loot table bug that was causing items to drop at unexpectedly low rates. This update is server only and requires no client side patching.
Later this week we are hoping to open up the new QA to anyone who has an active account. We will continue pushing content to QA throughout the month; see below for a teaser of what is to come.
Bug fixes / improvements:
The QA server is nearly operational. If all goes well we may be able to open this up later this week for a few days. A sampling of the current pending changes currently on QA, or making their way to QA this month:
To see a full list of what is currently in progress head on over to this thread. I will do my best to update this list on a weekly basis and highlight the differences from the previous week with bold font. Note that what is listed here and in the linked post is not necessarily everything that is currently in progress. Below I will highlight some of the upcoming changes in more detail.
Itemization V2 (ongoing)
We are in the midst of working through all items in the game to rebalancing their stats. The general goal is to allow for items to be more easily distinguishable from one another and "feel" like upgrades as you progress through them. This also allows for a more clear cut difference between items dropped from 1/2/3^ uncommon mobs, imbued, and rare items. A change of this magnitude will require quite a bit of time on QA but we feel that it will have an overall positive impact on the game.
Grimstone Canyon (QA soon)
A new level 30+ zone is currently working its way through the design phase and should be on QA within a week or two. This zone provides the perfect opportunity to test our new loading routine. If successful, we can begin to enable this tech for other scenes to reduce the overall size of future patches.
Technical Details:
Loot Table Fix
For the past several weeks the community has raised the alarm about our loot tables failing after a period of time; and I am both happy and sad to announce that they were right! Albeit only partially right - they were indeed failing but it had nothing to do with the passage of time. Let me explain...
Since this issue was reported I've been pouring over the code to try and figure out where the problem might be. The first step was reproducing the problem; which unfortunately was something we could not do locally. Our loot tables in the editor have a "Sample" button which allows us to take a large number of samples at once and examine the drop count of each item returned. The great thing about this button is that it uses the exact same code path as NPCs do when they generate their loot. Which means that looting an NPC X number of times is equivalent to hitting the sample button X times in the editor. After every report of broken loot we would inspect and test the item's loot table via the above mentioned sample button. Every single item we investigated returned the expected result on their respective loot tables. What made no sense was that things would work normally and suddenly stop later in the week. Nothing in the code depends on time or how many times a loot table has been sampled, so the idea that the loot table would just stop working made no logical sense. This means that you were all crazy and this was just RNG...right?
You see, as a developer, it is very difficulty to track down "nebulous" bugs. The feeling throughout the community was that something was wrong; but we had no hard data showing us that anything was actually wrong. All we had was anecdotal experiences that were varied wildly in the what/who/when/where department. Some things are meant to be rare (i.e. thief's blade), so some reports we chalked up to a bad streak of RNG. Other reports however, were a bit more concerning as their reported drop rates were much lower than what we expected (croc teeth). The first step in learning more about this problem was identifying specific items that were of concern, figuring out what they had in common, and running some experiments on the live server.
I had gotten a report about a specific torch dropped from a specific mob in Redshore's Exile Freehold. There were a few players roaming that zone for upwards of 12hrs/day which gave them a good general sense of what the drop rate was for that particular item. They kept me informed of the drop rate throughout the week and on Friday they reported that the torch had stopped dropping. I logged into the GM zone on live and spawned 100 of those NPCs to validate the drop rate of the torch - everything checked out as expected. This told me that the loot table was operational and any issues were unlikely related to the length of time a zone was up. Next I headed to the Exile Freehold and started spawning these mobs - low and behold a torch dropped! I continued spawning them until I hit 100 NPCs and was able to validate that the torch was dropping at the intended rate. Later that day the original reporter notified me that the torch was once again dropping. This is where I start to question my own sanity. Did my experiment somehow "reset" the loot table so that players were now getting the drop again? After pouring over the code again I convinced myself that this was not the case. Nothing in the code paths suggested that my manual spawning of NPCs should have had any impact on the actual results. Torches continued to drop up until patch day so I kept my ear to the ground for the next item I would run my experiment on.
During this time I have also been leveling my character, reached level 14, and had been assigned the croc teeth quest. This seemed like a perfect opportunity to get down in the trenches and see what the drop rate was for myself. Needless to say, I killed a few hundred crocs over the span of 3 days and was unable to obtain the quest item in question. After discussing the intended drop rate with Adric this is when I realized that something was indeed going on. To make matters even more confusing, Drindin was able to obtain the quest item after about 20 croc kills on my second day. This told me that something was broken, but not completely broken. And this is likely the reason why player reports have been so nebulous and varied. The loot tables have been gas lighting all of us!
Shortly thereafter a theory of what was happening hit me like a ton of bricks. In order to understand the issue one has to understand a little about the structure of our loot tables. To make Adric's life easier I developed something called a "LootTableVariant" which is basically an extension of another loot table. These are used in various ways throughout the game, but if we look at the croc example from above the loot table structure might look something like this:
Next we have to understand how loot tables are initialized. When they are first accessed, the loot table assigns normalized probabilities to each item, sorts by normalized probability, and then calculates a random number "threshold" for each individual item. It is important to remember that this process happens once and those values do not change after initialization; at least they aren't supposed to. Now here is where the problem comes into play. The LootTableVariants have to rebuild their entire table at run time since they are extending another table. In other words: - we want the new extended loot table to be independent from the original so we end up making a copy of the original before combining it with the extended items.
In code, these loot table entries are what we call a "class" and classes are what we call "reference types". All this means is that any code referencing that particular item is just looking at a specific chunk of memory. In other words: anyone making changes to that specific item would be making those changes for anything referencing that specific item. As it turns out, when building the loot table for each LootTableVariant it wasn't actually making a copy of the entry, it was instead just copying the reference to it. This meant that upon initialization of each LootTableVariant it was also changing the random number threshold of items inside of the base loot table.
Once I realized what was happening everything started to make sense. Why sometimes things worked for a while and stopped, or why it sometimes never happened at all. In this specific croc example the sequence of events that was causing the failure is as follows:
tldr: This week I identified and fixed an issue with loot tables that was causing some items to drop at drastically lower rates than we intended. Thank you all for continuing to bug us about it!
Known Issues:
This week brings a handful of critical fixes to the server. This includes temporarily rolling back Riposte damage so NPCs aren't one-shotting you, fixing a rare exception that could cause a zone to crash, and fixing a loot table bug that was causing items to drop at unexpectedly low rates. This update is server only and requires no client side patching.
Later this week we are hoping to open up the new QA to anyone who has an active account. We will continue pushing content to QA throughout the month; see below for a teaser of what is to come.
Bug fixes / improvements:
- Tech
- Fixed rare server-side exception when removing some buffs/debuffs.
- Fixed an issue with loot table initialization that was causing drop rates for some items to deviate from their intended rates (see below for more details).
- Design:
- Reduced Riposte damage. This is a temporary fix until we can properly separate NPC and Player riposte skills (soon).
The QA server is nearly operational. If all goes well we may be able to open this up later this week for a few days. A sampling of the current pending changes currently on QA, or making their way to QA this month:
- Bug fixes (including group invite fixes)
- Environmental prop fixes
- Texture fixes
- Creature audio normalization
- Riposte updates
- Dueling by death rolls
- Multiple new quests are on QA, but they will not be available as we don't want anyone spoiling the fun
- New level 30+ abilities
- New outdoor 30+ zone which will utilize new loading tech to reduce patch size
- Unity minor engine revision update (lots of bug and performance fixes)
To see a full list of what is currently in progress head on over to this thread. I will do my best to update this list on a weekly basis and highlight the differences from the previous week with bold font. Note that what is listed here and in the linked post is not necessarily everything that is currently in progress. Below I will highlight some of the upcoming changes in more detail.
Itemization V2 (ongoing)
We are in the midst of working through all items in the game to rebalancing their stats. The general goal is to allow for items to be more easily distinguishable from one another and "feel" like upgrades as you progress through them. This also allows for a more clear cut difference between items dropped from 1/2/3^ uncommon mobs, imbued, and rare items. A change of this magnitude will require quite a bit of time on QA but we feel that it will have an overall positive impact on the game.
Grimstone Canyon (QA soon)
A new level 30+ zone is currently working its way through the design phase and should be on QA within a week or two. This zone provides the perfect opportunity to test our new loading routine. If successful, we can begin to enable this tech for other scenes to reduce the overall size of future patches.
Technical Details:
Loot Table Fix
For the past several weeks the community has raised the alarm about our loot tables failing after a period of time; and I am both happy and sad to announce that they were right! Albeit only partially right - they were indeed failing but it had nothing to do with the passage of time. Let me explain...
Since this issue was reported I've been pouring over the code to try and figure out where the problem might be. The first step was reproducing the problem; which unfortunately was something we could not do locally. Our loot tables in the editor have a "Sample" button which allows us to take a large number of samples at once and examine the drop count of each item returned. The great thing about this button is that it uses the exact same code path as NPCs do when they generate their loot. Which means that looting an NPC X number of times is equivalent to hitting the sample button X times in the editor. After every report of broken loot we would inspect and test the item's loot table via the above mentioned sample button. Every single item we investigated returned the expected result on their respective loot tables. What made no sense was that things would work normally and suddenly stop later in the week. Nothing in the code depends on time or how many times a loot table has been sampled, so the idea that the loot table would just stop working made no logical sense. This means that you were all crazy and this was just RNG...right?
You see, as a developer, it is very difficulty to track down "nebulous" bugs. The feeling throughout the community was that something was wrong; but we had no hard data showing us that anything was actually wrong. All we had was anecdotal experiences that were varied wildly in the what/who/when/where department. Some things are meant to be rare (i.e. thief's blade), so some reports we chalked up to a bad streak of RNG. Other reports however, were a bit more concerning as their reported drop rates were much lower than what we expected (croc teeth). The first step in learning more about this problem was identifying specific items that were of concern, figuring out what they had in common, and running some experiments on the live server.
I had gotten a report about a specific torch dropped from a specific mob in Redshore's Exile Freehold. There were a few players roaming that zone for upwards of 12hrs/day which gave them a good general sense of what the drop rate was for that particular item. They kept me informed of the drop rate throughout the week and on Friday they reported that the torch had stopped dropping. I logged into the GM zone on live and spawned 100 of those NPCs to validate the drop rate of the torch - everything checked out as expected. This told me that the loot table was operational and any issues were unlikely related to the length of time a zone was up. Next I headed to the Exile Freehold and started spawning these mobs - low and behold a torch dropped! I continued spawning them until I hit 100 NPCs and was able to validate that the torch was dropping at the intended rate. Later that day the original reporter notified me that the torch was once again dropping. This is where I start to question my own sanity. Did my experiment somehow "reset" the loot table so that players were now getting the drop again? After pouring over the code again I convinced myself that this was not the case. Nothing in the code paths suggested that my manual spawning of NPCs should have had any impact on the actual results. Torches continued to drop up until patch day so I kept my ear to the ground for the next item I would run my experiment on.
During this time I have also been leveling my character, reached level 14, and had been assigned the croc teeth quest. This seemed like a perfect opportunity to get down in the trenches and see what the drop rate was for myself. Needless to say, I killed a few hundred crocs over the span of 3 days and was unable to obtain the quest item in question. After discussing the intended drop rate with Adric this is when I realized that something was indeed going on. To make matters even more confusing, Drindin was able to obtain the quest item after about 20 croc kills on my second day. This told me that something was broken, but not completely broken. And this is likely the reason why player reports have been so nebulous and varied. The loot tables have been gas lighting all of us!
Shortly thereafter a theory of what was happening hit me like a ton of bricks. In order to understand the issue one has to understand a little about the structure of our loot tables. To make Adric's life easier I developed something called a "LootTableVariant" which is basically an extension of another loot table. These are used in various ways throughout the game, but if we look at the croc example from above the loot table structure might look something like this:
Where the "croc loot table" represents a base, or "parent" loot table. This would contain items shared by all crocs such as worn teeth, reagents, etc. Then we have two "LootTableVariants" (rare and river) which essentially extend the base loot table. The river croc table adds the quest item of unbroken teeth, while the rare croc table adds rare items.
Next we have to understand how loot tables are initialized. When they are first accessed, the loot table assigns normalized probabilities to each item, sorts by normalized probability, and then calculates a random number "threshold" for each individual item. It is important to remember that this process happens once and those values do not change after initialization; at least they aren't supposed to. Now here is where the problem comes into play. The LootTableVariants have to rebuild their entire table at run time since they are extending another table. In other words: - we want the new extended loot table to be independent from the original so we end up making a copy of the original before combining it with the extended items.
In code, these loot table entries are what we call a "class" and classes are what we call "reference types". All this means is that any code referencing that particular item is just looking at a specific chunk of memory. In other words: anyone making changes to that specific item would be making those changes for anything referencing that specific item. As it turns out, when building the loot table for each LootTableVariant it wasn't actually making a copy of the entry, it was instead just copying the reference to it. This meant that upon initialization of each LootTableVariant it was also changing the random number threshold of items inside of the base loot table.
Once I realized what was happening everything started to make sense. Why sometimes things worked for a while and stopped, or why it sometimes never happened at all. In this specific croc example the sequence of events that was causing the failure is as follows:
- Someone loots a river croc --> RiverCrocLootTable is initialized, which in turn modifies the random number thresholds inside of the "CrocLootTable".
- All loot continues to drop normally from river crocs. Until...
- Someone loots a rare croc --> RareCrocLootTable is initialized, which in turn again modifies the random number thresholds inside of "CrocLootTable".
- Loot stops dropping normally for river crocs until a server reset. This is because the items from "CrocLootTable" are still in the river croc's combined table, but the thresholds have been changed for the base items.
tldr: This week I identified and fixed an issue with loot tables that was causing some items to drop at drastically lower rates than we intended. Thank you all for continuing to bug us about it!
Known Issues:
- Resource nodes may occasionally tell you "invalid target". This means that your client thinks a resource node is there but the server does not. We are still investigating the cause of this.
- Occasionally a loot roll may be skipped entirely and a carcass will be given. When this happens please provide any relevant details as we are still trying to track this one down.
- Deconstructing crafted items returns junk 100% of the time.
- Not all clothing/armor has visuals
- Dragging a group-nameplate in the group window will set the wrong pivot point for the group window and the whole thing will drag a bit weird.
- Typing chat text may occasionally become invisible. Unfocus your chat window and trying again a few times typically resolves this.
- Sometimes when you respawn you don't end up at the proper Hallow --> if this happens please report the position where you died via the "/report" command or "/debugposition" (if using debugposition you'll need to paste the value here)
- MacOS:
- Zooming the camera while holding the right mouse button can cause your chat window to scroll. This is not intended behavior.
- At some camera angles the sun may go dark. This should not happen frequently but is readily reproducible at some locations. Investigating a fix.
Last edited:
hopefully this means some of the crafted gear will be more useful now than just common drops. - looking forward for the QA server and having all these new things added soon!
