design What's the delay?

[Ueda]

Dear Ninja,

The next update for Nin Online has been awhile coming now, and we're also not sure when we'll be able to get it ready to launch. We're working almost every day of the week to try to make sure the update is the best it can be. But in the end, it's only really going to disappoint a large number of people because just by launching it because it's not like straight out content updates, we've been working on underlying systems a lot to get the game fully modernized.

We put time into the wrong things

In hindsight, a lot of what we've been working on is less important than it seemed. Like we re-programmed the entire game's saving to be Object-Oriented, in a production-level cloud database, MongoDB Atlas, but in the end, it's still going to be entirely the same for the end user. So it's a shame, it's something I believed was going to be a seamless transition, but took weeks out of our time to resolve all the problems, and we might still be having some.

And again, in hindsight, this should've taken a backseat to working on things people care about or would directly affected their gameplay. Cloud saving is nice to have, and it brings about opportunity for stuff like a Web frontend for Nin, like imagine being able to display your characters on the forum, or edit their name from the web, manage your items etc.

We rewrote the login system to use oAuth, a clean login standard that means no more entering your password every time you want to login too. It also simplifies our server's job of checking MySQL tables to check if a player is banned because oAuth handles all that on the client and web side. Logins should also just be faster in general now, as we've rewrote a lot of the underlying login code as well.

We updated the server to work with .NET 5 instead of .NET Framework. It's basically Microsoft's latest development framework. Which sounds like a small change, but .NET Framework is ancient technology at this point. .NET 5 allows us to modernize the server even further and export it as a containerized server for hosting services like Heroku. This also means that the server is now a console application, and cross-platform (we can host it on Linux servers which are cheaper because Linux is free and Windows Servers need licenses, Linux servers are also more easily available). Along with this, we also halved the server loading/launching time. So less down time between server restarts.

Wolf rewrote the packet handler thread in this update, it should shave off about 20ms off larger packets. We updated our networking library to use it's latest version, this was an attempt to fix some low-level bugs with disconnection we were having.

We replaced the compression library from Zlib to LZMA, which fixes a lot of compatibility issues players were having with the new client and potential crashes. We also managed to get map packets and map loading down a lot with better compression along with reducing the amount of packets needed to be sent on login by almost 10 fold. So faster logins, better map loading times. But in order to do this, it meant we had to deprecate the old client. It's a weird situation. In order to deprecate the old client we needed to make this change, but Zlib was also making it so some people couldn't move to the new client.

We have the player housing update, furniture update and corps update in this too one patch too.

We also used MongoDB to replace an entire component of our server architecture, our authentication. It's now an entirely passive web-based solution. Servers register to an online database instead of sending that data to the authentication server to do the central server selection stuff. As such, we also had to make it so the server does the saving directly to the cloud instead of the authentication server. So a lot of these things were interconnected, and had to be all done at once.

This directly caused us to stack update after update into one patch, and it became harder and harder to push one patch without issues. With all the major architectural changes and low-level code rewrites, a lot of things broke. Because of the nature of the changes we were making, we also couldn't make them independently of each other, so it had to be one big update or nothing.

There's also a hundred other things I've left out, including reworking movement on the client side to animate smoother. These things don't make the patch any more complicated, so it's not worth mentioning.

Hectic month of testing with BR Server

We started testing the latest patch around a month ago on the Brazil Server now. During that time, we've fixed issue after issue, and some of the "fixes" were also entire rewrites, which caused more issues. It's been stressful, we've had days spent just fixing login issues, and days spent just fixing saving.

We've had data corruption issues, where players accounts and all their data become too messed up to fix, and we've had to do hacky fixes for them like restore partially their data from old backups. We've had a period where the same player couldn't login for a span of days, and we realized the server wasn't disconnecting players properly. Stressful stuff.

We've had issues where resetting characters wasn't working and worst yet, it created a duping exploit. We've had more of these issues in a span of a month, than we've had over the last 3 years of development.

We pushed two new major features, one is currently being used actively, and that also came with it's own issues, but was a lot easier to fix than the issues that came from major architectural changes. Though some have been incidental to them.

We've had issues where players were logging in as other players when they tried to login at the same time.

We've had players spamming us telling us we're idiots and we should just put back the old client, we've had players spamming us telling us to revert to the old server, to stop using them as a test server, all while our full-time job had become fixing issues for a month. It's been stressful, for both me and Wolf.

Side note: Whenever Brazil players complain about the issues happening, we sympathize with them, but we also think about how up until now, they've only been getting updates that have been screened and tested by the NA/Global server (you guys) and so they're fortunate to not have been the guinea pigs up until now haha

Irreversible Update

The worst part in all this was the difficulty in which the reversal of this update would've been. In order to update Brazil server, we needed to move all of the existing data into the cloud into a slightly different format. Which meant if we want to reverse the update, we would need to build a software to convert the data back into our old format as local JSON files or rolling back players a month.

So it's been a go hard or go home month. There were times where I'm so tired I wanted to reverse the entire update and call it quits (this also happened just yesterday) but I had to force myself to look through code and fix it instead, despite being so sick of it.

This is going to be even more so irreversible for the main server, because in order to update the next patch, we have to update the forum software, and that's one-way. So we're going to have to be even more careful with this server, which is why we're taking our time and making sure everything is good to go.

The worst part

I'm also stressing out about how this update won't bring enough content to make players happy, or bring gameplay value to people. It's very painful, knowing this, because we've been working so hard on this, but I know in the end, this is more for the sake of the game in the long term, and also some of it is just wasted effort. Most people won't care that the game server runs slightly smoother and that login is faster, more convenient etc. Nobody is going to care that player data is now stored in a cloud database. This update has been months of work now, but I'm just waiting for the criticism on how it's a shit update etc. It's going to happen, and I'll understand why, but it'll still be hurtful.

This update when it takes into account how much work has gone into it by two of us, thousands of hours of work, which means tens of thousands of dollars worth of work. It's not funny for me, it's very painful. And I agree with players, is it really necessary? We could've done without. I could've just not have a full-time programmer, and I could've just focused my time on working on stuff people care about like adding more Jutsu. This has been the hardest months of development since maybe 2017 when the server was unstable, crashing every 1-4 hours, and it's self inflicted.

The bright side

The bright side is that things are looking much better, there seems to be some rare bugs that happen, but it's stable for the most part now. One guy claims he managed to login multiple times in a row without it saving his progress, which we're looking into. And there's some others who seem to not be able to login sometimes. We're looking into that too.

The reception of the update has been a lot better, after holding a tournament yesterday, Kenock (Brazil server admin) remarked that the server felt smoother than ever with a large number of people on the map, and it was a high activity period of Brazil server as well, so there were lots of players online in general. Brazil server host is also notoriously bad (despite being super expensive), so if it runs on Brazil's server well, it's going to run even better on NA's.

We don't know when we'll be confident to launch this update on the main server yet. Because if anything messes up, we'll be running heroics, fixing corrupted data, or salvaging the situation, pushing updates on two servers instead of one. This is another reason why we regret so much making this update so major (on an architectural level).

We did also have to do a lot of this in order to deprecate the old client. We did spend some time making sure the new client is ready to completely replace the old client, and that anyone will be able to use it. So we haven't had anyone on Brazil server that can't use the new client completely anymore...

Lastly, I'd just like to apologize in advance if what we've worked on doesn't please you. I know, it's not what anyone wanted. But we've been doing what we thought was important. We've re-evaluated what's more important going forward and will focus on bringing content updates here on out. It's both easier for us, and more fun for you.

Regards,
Ueda

EDIT: Just thought of something funny. Maybe the reason we worked on Cloud saving is because everyone kept asking "Cloud when?" and we interpreted it wrongly  

EDIT2: Some of the low-level changes we made include rewriting the game's method of calculating time, and the spell buffer. This could've been causing issues with sometimes your jutsu not casting when you press it, and small things like that - that maybe some players will notice.

[Average]

The background update. The one no will see, and few will feel. A minority I might be, thank you Ueda for doing what you felt need be done.

[Slifer]

/y

[Erox]

I've been on both sides of this (Staff Chat/Private Messages and NinGen/InGame) since the last major update and I can honestly say the long term pros outweigh the short term cons, No matter the backlash that comes from this I got your back!

[Nefarious]

You cannot grow if you do not have a solid foundation. 

I'd rather get the groundworks for all these new features laid out and established before adding on unstable content that would be more of a pain to fix than to just wait for.

Look what happened to CyberPunk when it got rushed. I'd rather have a stable game with a lot of room for growth than an unstable game with a lot of buggy content

[Imhotep]

The new client improved my quality of gameplay 10-fold. An update like that is necessary, and I don't think the team can be faulted for continuing down that line. It's arguable that it was even a mistake, but if it was - allow yourself a misstep, especially if others won't.   

That being said, content is king. The community, myself included, will be glad to know it's on the forefront again moving forward.

[Sushi]

Dont take us wrong Ueda, most will still make fun of you for not doing anything new in the game, its just running meme now, but i believe deep down, and mainly for me I know its like that, most people know that you are doing what you can and I appreciate the work as I was just small when I had dream of making my own game and I didnt yet because all this work around. Dont overwork yourself, even if we make fun of you, we care about you. 
ps: are there advanceds in the new update? 

[Hanzaki]

I might be a new player, but I have been playing games for a few years now. Caring about your community (or what is being said) is something important, and I respect you a lot for that, but that very community also needs to understand that game development also includes what you have done: the incoming update is as much important as the ones that add contents. 

That being said, you are doing an incredible game, and I don't regret spending hours on what you've created. 

[Emiya]

I firmly believe that putting priority on server stability is dozens of times more valuable than working on any other updates currently.

Even if most people won't be able to see or appreciate the fact that no updates were made because our developers had silently been working in the shadows spending tons of hours of dedication and hard work into improving the game. At the end of the day, while a portion of the community might have their complaints in short-term, it is going to make a lot of people happy in the long-term. 

Personally I've always wanted to see the focus be put in giving players somewhat of equal chances in terms of map switch loading, or running speed for example. I think the advantage given to users with higher fps and lower ping is significantly huge and frustrating to see. I've seen players refuse to use the new client because of the low speed of map switch loading. It has been bugging me personally as well.

I also get that some people aren't happy to be used as experimental test subjects on the BR Server. But to me it looks like you guys are trying to build a rocket booster on a brand new car/engine that's still in development. And I don't see an issue in putting it out for a test drive and observe your current progress in real-time in the game. Some would still prefer the old car cause the seats feel smoother, but you guys are trying everything you can to make v2.0 even smoother than that. What is there not to appreciate?

I know the "Rory's vision" has been kind of a meme, but I think you have a very clear understanding on where to put your focus on right now. I've watched this game grow from its infant stage from 2013 all the way up until now. You wouldn't have been half as successful if your vision wasn't as good as it is. Is it perfect? No of course not because we all have our flaws. But you do possess the responsibility as you take accountability with your actions, and make changes whenever you note mistakes that you didn't see at first. You also listen to the community really well and made improvements based on the experiences of what you think is the best course of action. 

The game didn't just miraculously become successful. You put your heart and soul into this master piece of work. You've always shown great professionalism and took your project as seriously as you possibly can. On top of that, you've also given clear transparency on what it is that you've been working on. And for that I'm incredibly grateful and happy that I've been part of this ride for so long as I'm eagerly awaiting where the road takes us next in the future. 

I'm wishing you nothing but the best, especially cause I can't imagine how much stress this has brought to you. But I'm confident that despite all of your sleepness nights, it's going to pay off greatly at the end. Make sure to reward yourself and take a good well deserved break too, cause you deserve to allow yourself to de-stress and recharge afterwards. You've earned every single dime with the fruits of your labour. And we're able to see you invest your time and money to build things to the next level. As a result you've allowed us to enjoy spending our time here at the expenses of your hard work.

So thank you for everything and stay freaking awesome.

[Etsu]

Back end changes are mostly underestimated because no one see them but everyone need em. From my point of view this update looks great and will improve overall quality of everything published after it. I can’t imagine how work on something what was firstly created in vb6 can look like but I know that most of people, including me, would already give up. These upgrades are amazing and I can’t wait to see a final results of your work guys!
Take your time and deliver finished product

[Desecration]

Thank you so much for your hard work Ueda and all of the nin staff, I honestly rather having a game that runs/ performs well than new in game content.  Even tho I'd love new content I know that I cant enjoy that content if my game doesnt run well, so once you guys do what you believe will support this game long term I'll support it. Another thing I am certain the majority of the community appreciates what you do for this game so I gotta say to please try to focus on them, we love you guys for your hard work. Keep it up  

[Soshiki]

Thanks man i really appreciate all the hard work (not to mention countless hours) your putting into the game so it can run smoother

P.S.
I don't care if there is no new content i'm still new to the game so i'm still getting all that content

[Accelerator]

Thank you for being transparent with us.

Hopefully we can finally do Joint CE's if the server is very stable!

[Heavens Angel]

6 hours ago, Ueda said:

 

EDIT: Just thought of something funny. Maybe the reason we worked on Cloud saving is because everyone kept asking "Cloud when?" and we interpreted it wrongly  

 

ROFLMFAO this one got me good  

[Sn1per Old]

Faster logins and map switches are a huge thing and also having a solid foundation to work on is a lot better than adding new jutsu and new crap that people will complain is too op within hours or launch, there will always be people complaining but in the long run you are doing things that are better for the game as a whole. Keep up the work and ty for this insight on everything that has happened is has made a lot of us aware of what is happening and that we shouldn't be shit talking the developers or the game.

 

[Roei]

(Erox read the message before, and don't touch it).

16 hours ago, Ueda said:

This update has been months of work now, but I'm just waiting for the criticism on how it's a shit update etc. It's going to happen, and I'll understand why, but it'll still be hurtful.

Do not get me wrong Rory, I think that this Update overall will help Nin in the long run, stability is an important thing.
However as much as I don't wanna make you feel bad, there is a lot of criticism which you do need to acknowledge, these critics matter and people say these because they care about Nin. and while they may sound hurtful to you, they are necessary to take care of if you want Nin to be better.

While you all been working on hard  and focusing on the development part on your own you left out the community. 
First, there is no player transparency, people would like you to involve them in the decisions and the actions before you're done doing them.
People feel lost and do not know anything about the development part, and while you may think you're making the best decisions do not be upset to get these reactions afterwards.
Second there are hundreds if not thousands of amazing people who can and want to help and contribute to Nin online, they are a great solution to a lot of the problems regarding the lack of content made. 
You often times forget this but Rory this game is community focused, and your players do not get their voice heard and that is a fact, say whatever you want posts of discussions and suggestions about important things people care about go unnoticed or not replied.
(these are obviously just a few of the critics but it's good to start there first)
I suggest you'd start to involving the community a lot more with the development of the game, and do not let critics of any kind to stop you from listening to the players.

And this is a quick message to all the Nin players, if you all want the game to be better, and not just rely on hope to get what you want, take an action make a change, show them that you matter, cause otherwise this behavior will just continue, and you being all nice and good kids every time does nothing for you.

Your opinions matter, don't let anyone tell you otherwise.

As some of you may know, I as player am done with Nin Online, but I want to believe there is hope that the struggles we had to deal with won't continue to be a problem for those who still play and care about Nin Online.