Heroes of Might and Magic Community
visiting hero! Register | Today's Posts | Games | Search! | FAQ/Rules | AvatarList | MemberList | Profile


Age of Heroes Headlines:  
5 Oct 2016: Heroes VII development comes to an end.. - read more
6 Aug 2016: Troubled Heroes VII Expansion Release - read more
26 Apr 2016: Heroes VII XPack - Trial by Fire - Coming out in June! - read more
17 Apr 2016: Global Alternative Creatures MOD for H7 after 1.8 Patch! - read more
7 Mar 2016: Romero launches a Piano Sonata Album Kickstarter! - read more
19 Feb 2016: Heroes 5.5 RC6, Heroes VII patch 1.7 are out! - read more
13 Jan 2016: Horn of the Abyss 1.4 Available for Download! - read more
17 Dec 2015: Heroes 5.5 update, 1.6 out for H7 - read more
23 Nov 2015: H7 1.4 & 1.5 patches Released - read more
31 Oct 2015: First H7 patches are out, End of DoC development - read more
5 Oct 2016: Heroes VII development comes to an end.. - read more
[X] Remove Ads
LOGIN:     Username:     Password:         [ Register ]
HOMM1: info forum | HOMM2: info forum | HOMM3: info mods forum | HOMM4: info CTG forum | HOMM5: info mods forum | MMH6: wiki forum | MMH7: wiki forum
Heroes Community > Heroes 7 - Falcon's Last Flight > Thread: An Empirical Study of MMH7 Memory Consumption and Time Complexity
Thread: An Empirical Study of MMH7 Memory Consumption and Time Complexity This thread is 5 pages long: 1 2 3 4 5 · «PREV / NEXT»
verriker
verriker


Honorable
Legendary Hero
We don't need another 'eroes
posted August 24, 2016 12:28 PM

Oakwarrior said:
You... do realize that you made a render stress-test and not a memory stress-test, right?


how's it going there, did you find the skills and resources yet to patch up the nasty memory leaks and dreadful AI of H7 lol
____________

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Bitula
Bitula


Known Hero
posted August 24, 2016 12:28 PM

Galaad said:
Quote:

This is a universal problem affecting all games.



The same way it is affecting h7?


Well who knows... We have a very low amount of sample. You and Stevie or what… Others have low end.

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Oakwarrior
Oakwarrior


Famous Hero
posted August 24, 2016 12:38 PM
Edited by Oakwarrior at 12:47, 24 Aug 2016.

verriker said:
Oakwarrior said:
You... do realize that you made a render stress-test and not a memory stress-test, right?


how's it going there, did you find the skills and resources yet to patch up the nasty memory leaks and dreadful AI of H7 lol


No we're too busy making sure the MP always goes out of sync at least once per game..




LizardWarrior said:
That's why I started allocating long doubles continously on the heap


You will only use up the memory it takes for long doubles. Which is 8 bytes. Even if you do it every tick it will take you a long time to get a result. Lets say you load the Ancient Behemoth's mesh and textures into the memory. That's about 37 megabytes.

Just to give you an idea, the visuals data for just our creatures is around 6Gb. That's stuff like meshes, textures, some particles (which also use textures and meshes). We use about 25gb for the entire visual package of this game. So in the worst case, if you have a big map with all factions represented and every creature recruited or in the map, that would mean your system would need more than 25 gigs of RAM to run the game without any modification to the assets. THAT would be a stress test

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Stevie
Stevie


Responsible
Undefeatable Hero
posted August 24, 2016 12:47 PM

I see what you're trying to say, but you're opening another can of worms by saying it. 25 GB of RAM to run an epic map, excessive to say the least. How come no one predicted this outcome to try and steer away?
____________
Guide to a Great Heroes Game
The Young Traveler

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
frostymuaddib
frostymuaddib


Promising
Supreme Hero
育碧是白痴
posted August 24, 2016 12:49 PM

The silence has been broken!
____________
"Occam's shuriken: when the answer is elusive, never rule out ninjas." -- Dr. Gordon Freeman (Freeman's Mind)
"lol" -- VERRIKER VON ERWINSSEN

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
LizardWarrior
LizardWarrior


Honorable
Legendary Hero
Age of erwins is over
posted August 24, 2016 12:56 PM

Not quite, I push up 100.000 long doubles every second in a dynamic array, for every mesh I spawn. The function is continuously ticking, I'm using an array of an array of long double pointers as a member of the class, so every 0.1 seconds, an array of 1000 long doubles is formed and is pushed upon the member array. So beside that I allocate 100ks of long double every second (as every mesh will allocate 1k long double), it will also push them on the dynamic array, thus reallocating memory to resize the array every 0.1 second. Assuming the TArray has a size_t type maximum size, that means up to 2^64 arrays of 1k long doubles per 3d mesh spawned. So 2^64 * 10^3 * 8 bytes maximum per actor spawned, not taking into consideration the implementation value of TArray classes, and accounting that every actor has a spawn delay of 0.1 - 0.2 seconds, then it goes pretty far.

Also, correct me if I'm wrong, but I'm sure that the engine is smarter than that, first if you get an whole overview of the whole map, it will be rendered from quite afar, so the level of detail comes in to play, then at a camera distance where the level of detail will be maximum, then it will be pretty close, thus the engine won't render what it can't see.
____________

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Oakwarrior
Oakwarrior


Famous Hero
posted August 24, 2016 12:58 PM
Edited by Oakwarrior at 13:04, 24 Aug 2016.

Stevie said:
I see what you're trying to say, but you're opening another can of worms by saying it. 25 GB of RAM to run an epic map, excessive to say the least. How come no one predicted this outcome to try and steer away?


Because you don't need 25GB of RAM to run an epic map, obviously. H7 takes 17Gb storage space when installed. But we have 25Gb of visuals in there. Let that sink in for a moment. That's just the raw data, and assuming that every single asset is loaded in. Ever notice texture/mesh popping in games? That's what happens - assets get loaded in on demand (or different versions of it's LoD). And definitely not in their raw form. It's quite visible in Diablo 3 for example or, more recently, No Man's Sky. H7 has it as well.

Oh and don't worry about the silence, this is purely personal
I just couldn't bear to read this and not react to it because of how well you guys demonstrated how much you think you know about game development.



LizardWarrior said:
Not quite, I push up 100.000 long doubles every second in a dynamic array, for every mesh I spawn. The function is continuously ticking, I'm using an array of an array of long double pointers as a member of the class, so every 0.1 seconds, an array of 1000 long doubles is formed and is pushed upon the member array. So beside that I allocate 100ks of long double every second (as every mesh will allocate 1k long double), it will also push them on the dynamic array, thus reallocating memory to resize the array every 0.1 second. Assuming the TArray has a size_t type maximum size, that means up to 2^64 arrays of 1k long doubles per 3d mesh spawned. So 2^64 * 10^3 * 8 bytes maximum per actor spawned, not taking into consideration the implementation value of TArray classes, and accounting that every actor has a spawn delay of 0.1 - 0.2 seconds, then it goes pretty far.

Also, correct me if I'm wrong, but I'm sure that the engine is smarter than that, first if you get an whole overview of the whole map, it will be rendered from quite afar, so the level of detail comes in to play, then at a camera distance where the level of detail will be maximum, then it will be pretty close, thus the engine won't render what it can't see.


100k long doubles per second means you're allocating around 6.4Mb, per tick, right? In that case you would have crash a few seconds after startup. Also note that there's a difference between allocating and reserving.

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
JollyJoker
JollyJoker


Honorable
Undefeatable Hero
posted August 24, 2016 01:07 PM

Oak, I cannot follow you: EITHER you failed to optimize the process you are so pround of, resulting in a memory leak; OR the memory leak is unavoidable with that amount of data on demand - then you shouldn't have put that much graphics data into it in the first place, because it's unreasonable (and also counter-"strategic", since the most detailed view has the least strategic or tactical value).

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
frostymuaddib
frostymuaddib


Promising
Supreme Hero
育碧是白痴
posted August 24, 2016 01:10 PM
Edited by frostymuaddib at 13:13, 24 Aug 2016.

Oakwarrior said:

I just couldn't bear to read this and not react to it because of how well you guys demonstrated how much you think you know about game development.


I never said that I know game development I just tested the game (loading times, startup times, AI time) on my PC, and posted the results. Based on other players, it seems that this behaves very differently and that it depends on the pc - some are lucky to run the game smoothly, some are not. For me, that tells that the game was not tested enpough (confirmed by number of bugs) or that it wasn't tested on different machines. Just to be clear, I'm not attacking you personally, I'm just sick of seeing things similar to H6 lifecycle happening again with H7!

I don't know if you've played H6, but there was (is) infamous black screen bug: some people just couldn't start the game (me included) after certain patch. And let us all be fair: you released bug ridden unfinished game (is it ubi's fault or limbic's it doesn't matter in the end). You do not need to be genius or game developer to see that

Another thing: I really wanted and hoped that H7's release went smoothly after the H6 fiasco. I really wanted that, as like everyone else here, I love Heroes series. What happened pissed many off. Even I lost my nerves in the end, despite not owning the game, as I don't want my favourite game series to be treated like that.
____________
"Occam's shuriken: when the answer is elusive, never rule out ninjas." -- Dr. Gordon Freeman (Freeman's Mind)
"lol" -- VERRIKER VON ERWINSSEN

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
LizardWarrior
LizardWarrior


Honorable
Legendary Hero
Age of erwins is over
posted August 24, 2016 01:13 PM

I'm not saying that I'm better than you, I stated in my first post that I'm a noob in game programming, but the worse the programmer, the more I should screw it up, no? If I hit it up to 100k per second, then it just goes up fairly fast to about 4 GB and I don't dare let it more. That was just an example to see how to match the h7 memory consumption. I admit that in the 1st post I didn't let it load up and just took a screen. Memory usage matching h7's was at about ~1000 actors crunching numbers. Please don't take it personal, I got nothing against limbic, even if you don't admit it's pretty evident that ubisoft kinda left you under-budgeted and with not enough time, I just enjoy these discussions, because even if we baited people a little, there's something to learn out of it.
____________

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Oakwarrior
Oakwarrior


Famous Hero
posted August 24, 2016 01:16 PM
Edited by Oakwarrior at 13:33, 24 Aug 2016.

JollyJoker said:
Oak, I cannot follow you: EITHER you failed to optimize the process you are so pround of, resulting in a memory leak; OR the memory leak is unavoidable with that amount of data on demand - then you shouldn't have put that much graphics data into it in the first place, because it's unreasonable (and also counter-"strategic", since the most detailed view has the least strategic or tactical value).


I'm not talking about H7. I'm talking about the test that was made here. The purpose of which I have trouble understanding myself..

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Stevie
Stevie


Responsible
Undefeatable Hero
posted August 24, 2016 01:17 PM

Oakwarrior said:
I just couldn't bear to read this and not react to it because of how well you guys demonstrated how much you think you know about game development.


Em... right back at you? Remember, the performance of your product left much to be desired that it instilled the curiosity of community members here to put it to the test. As players they're not required to have dev knowledge, some other people responsible for coding and compatibility are.
____________
Guide to a Great Heroes Game
The Young Traveler

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Bitula
Bitula


Known Hero
posted August 24, 2016 01:21 PM

JollyJoker said:
Oak, I cannot follow you: EITHER you failed to optimize the process you are so pround of, resulting in a memory leak; OR the memory leak is unavoidable with that amount of data on demand - then you shouldn't have put that much graphics data into it in the first place, because it's unreasonable (and also counter-"strategic", since the most detailed view has the least strategic or tactical value).


A memory leak is simply forgetting or failing to de-allocate a previously allocated object which is by design meant to be temporal. It has nothing to do with the size or amount of data or with lack of optimization. In other words: it is a bug.

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
verriker
verriker


Honorable
Legendary Hero
We don't need another 'eroes
posted August 24, 2016 01:30 PM

Oakwarrior said:
I just couldn't bear to read this and not react to it because of how well you guys demonstrated how much you think you know about game development.


I take no offense from that remark myself, but since you were once in the community yourself, I hope you realize it's comments and actions smelling of a certain hubris like this, which have painted a less than flattering picture of you guys (Limbic),

it is a bit rich after all, let's have some perspective, you are basically an intern who was able to got into H7 development because you were in the VIP program and Limbic needed the cheapest people available, you have developed only one commercial game in your life as far as I know (which has been a severe critical and commercial failure), your studio has accrued more professional black marks from that process than I can list here (including sabotaging your own publisher and VIP group with leaks), and the person you are addressing has created many solutions to basic technical problems with your game you swore were impossible to solve, but you will still come in here and say, others don't have credentials to talk about game development lol

where are your credentials, what are your achievements, lol
____________

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Oakwarrior
Oakwarrior


Famous Hero
posted August 24, 2016 01:34 PM
Edited by Oakwarrior at 13:34, 24 Aug 2016.

LizardWarrior said:
I'm not saying that I'm better than you, I stated in my first post that I'm a noob in game programming, but the worse the programmer, the more I should screw it up, no? If I hit it up to 100k per second, then it just goes up fairly fast to about 4 GB and I don't dare let it more. That was just an example to see how to match the h7 memory consumption. I admit that in the 1st post I didn't let it load up and just took a screen. Memory usage matching h7's was at about ~1000 actors crunching numbers. Please don't take it personal, I got nothing against limbic, even if you don't admit it's pretty evident that ubisoft kinda left you under-budgeted and with not enough time, I just enjoy these discussions, because even if we baited people a little, there's something to learn out of it.


I wasn't trying to assert myself or anything. I respect you and what you do and you are basically the only person on this board I know that actually is genuinely curious on how games work and also the only person who is on the right track to find out. The rest is just mindless circlejerk over arbitrary numbers that hold little or no actual value (just to say, the original post by frosty does actually provide interesting and relevant metrics).

I even have trouble understanding what you guys mean with memory leak. If a game takes 5gb of memory, and then after advancing in it, taking into account that some changes will be persistent and then takes 6gb, that is not a memory leak. A memory leak for me means a constant and increasing (either linear or exponential) consumption of memory, that has no end and will run till the application crashes or becomes otherwise unuseable, like if it hangs. Fluctuations in memory usage are common, though.

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
LizardWarrior
LizardWarrior


Honorable
Legendary Hero
Age of erwins is over
posted August 24, 2016 01:37 PM

Thank you for your kind words. As about the memory leak, multiple people complained that the game starts building up memory the longer they play, this isn't just about increasing numbers in computations and having more factors in play, some even said that it doubles its memory usage after a few hours, I'm pretty certain there is a memory leak there. It happened to me too, but with the editor, it just starts pilling up memory the longer you use, even if you do nothing. I'd appreciate if you can bring up this problem with the team and fix it somehow.
____________

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Oakwarrior
Oakwarrior


Famous Hero
posted August 24, 2016 01:37 PM
Edited by Oakwarrior at 13:41, 24 Aug 2016.

verriker said:
where are your credentials, what are your achievements, lol


Right back at you, son.

LizardWarrior said:
Thank you for your kind words. As about the memory leak, multiple people complained that the game starts building up memory the longer they play, this isn't just about increasing numbers in computations and having more factors in play, some even said that it doubles its memory usage after a few hours, I'm pretty certain there is a memory leak there. It happened to me too, but with the editor, it just starts pilling up memory the longer you use, even if you do nothing. I'd appreciate if you can bring up this problem with the team and fix it somehow.


I've seen these posts myself and we've used numerous different set-ups and we're unable to reproduce it to that extent. I don't know if this is a residual issue from some pre-patch times, but it's basically not happening on a grander scale, lets say. Software works slightly differently depening on the machine its run on due to, well, hardware. No-one can test against all the hardware combinations in the world, so devs generalize. Fringe cases are likely to not be solved simply because they cannot be reproduced. When it comes to the editor, it's just the sheer amount of packages you're loading in. H7 is super data-heavy and thus also creates a great deal of load on the editor since UDK does not think that unloading assets once you don't need them anymore is really a necessity

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
frostymuaddib
frostymuaddib


Promising
Supreme Hero
育碧是白痴
posted August 24, 2016 01:39 PM

Oakwarrior said:
The rest is just mindless circlejerk over arbitrary numbers that hold little or no actual value


'circlejerk' is becoming very popular term by the ones who defend H7

Oakwarrior said:

(just to say, the original post by frosty does actually provide interesting and relevant metrics).



Thx. In a way, that was one of the goals. I'd love to see H7 optimised (this is not sarcasm, I really want that).

Oakwarrior said:

I even have trouble understanding what you guys mean with memory leak. If a game takes 5gb of memory, and then after advancing in it, taking into account that some changes will be persistent and then takes 6gb, that is not a memory leak. A memory leak for me means a constant and increasing (either linear or exponential) consumption of memory, that has no end and will run till the application crashes or becomes otherwise unuseable, like if it hangs. Fluctuations in memory usage are common, though.


I agree with this. And to be honest, my hour long test was not enough to detect memleaks. What I noticed however is a trend of linear growing. If I had time and will I would test tthe game longer, alas I don't have. Another thing that stopped me were crashes...
____________
"Occam's shuriken: when the answer is elusive, never rule out ninjas." -- Dr. Gordon Freeman (Freeman's Mind)
"lol" -- VERRIKER VON ERWINSSEN

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Oakwarrior
Oakwarrior


Famous Hero
posted August 24, 2016 01:46 PM

frostymuaddib said:
Oakwarrior said:
The rest is just mindless circlejerk over arbitrary numbers that hold little or no actual value


'circlejerk' is becoming very popular term by the ones who defend H7

Oakwarrior said:

(just to say, the original post by frosty does actually provide interesting and relevant metrics).



Thx. In a way, that was one of the goals. I'd love to see H7 optimised (this is not sarcasm, I really want that).

Oakwarrior said:

I even have trouble understanding what you guys mean with memory leak. If a game takes 5gb of memory, and then after advancing in it, taking into account that some changes will be persistent and then takes 6gb, that is not a memory leak. A memory leak for me means a constant and increasing (either linear or exponential) consumption of memory, that has no end and will run till the application crashes or becomes otherwise unuseable, like if it hangs. Fluctuations in memory usage are common, though.


I agree with this. And to be honest, my hour long test was not enough to detect memleaks. What I noticed however is a trend of linear growing. If I had time and will I would test tthe game longer, alas I don't have. Another thing that stopped me were crashes...



Circlejerk is popular mostly because it illustrates the situation quite well

In terms of optimization I'm not fully sure what you mean by this. Do you mean the FPS, memory usage or loading times? These are the three big performance metrics for Heroes.

To chime in again on the tests - everything has been tried: soak tests (just leaving the game running for ages and then going back to it), stress tests (worst case scenario testing), longetivity tests (to see if memory usage and FPS decline over time), and several more "fringe" test cases as well (with specific interactions for example). There is no hard evidence we have about a proper, full-on memory leak that we can follow and thus finding the issue (if there even is one) is incredibly difficult. I've had this issue with my own projects in UE4, but there is always a breadcrumb trail for a memory leak. It might be hard to find but there is one.

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
verriker
verriker


Honorable
Legendary Hero
We don't need another 'eroes
posted August 24, 2016 01:52 PM

Oakwarrior said:
Right back at you, son.


if expressions of contempt for your audience are really the only rejoinders you have in your arsenal, I honestly have to say I wish you and your studio ill tidings in turn, and that the failure of Heroes 7, can only be the chickens coming home to roost (while, believe it or not, I felt legitimate pity for Limbic before) lol
____________

 Send Instant Message | Send E-Mail | View Profile | Quote Reply | Link
Jump To: « Prev Thread . . . Next Thread » This thread is 5 pages long: 1 2 3 4 5 · «PREV / NEXT»
Post New Poll    Post New Topic    Post New Reply

Page compiled in 0.1596 seconds