CustomTexture: prefetch all available textures #2162

degasus · 2015-03-01T22:53:37Z

This PR tries to prefetch and to decode all available custom textures if possible. This behavior can be enabled with a GUI option.
The goal is to reduce the stuttering on png decoding while playing, but it will require lots of memoy. eg the biggest texture pack I've found is the xenoblade one which requires additional 4GB. As this may crash smaller systems, the prefetcher will abort if more than half of the physical RAM is used.

degasus · 2015-03-01T22:55:02Z

@BigheadSMZ Do you want to try the new option? It shall load all custom textures non-blocking (at least a bit) on startup.

magumagu · 2015-03-01T23:03:02Z

Why is the prefetcher on a separate thread? Does it really take so long that we can't block startup on it? Would parallelizing it help?

Could we maybe consider adding support for some format that loads faster instead (like .dds)?

Also, how much memory are we talking about? Gigabytes?

degasus · 2015-03-01T23:06:11Z

Here, it's about 35 seconds. So it is worth in my opinion.

Parallelizing would help, but soil isn't thread safe as written somewhere as comment...

.dds is supported by soil already.

The current xenoblade pack + environment are 3.6 GB, so it should never ever be enabled by default.

magumagu · 2015-03-01T23:06:40Z

Oh, another alternative: maybe we could track textures which are used together, so when one texture is used, we prefetch the others which are expected to be used soon.

magumagu · 2015-03-01T23:11:01Z

.dds might technically be supported, but it would be a heck of a lot faster if we could just memmap() the file and pass the pointer directly to the backend.

mimimi085181 · 2015-03-01T23:17:03Z

How fast do the textures load if they are the in the cache of the OS? I mean like, copy the textures from Dolphin's folder to anywhere, and then start Dolphin and let it load the textures?

Source/Core/VideoCommon/HiresTextures.cpp

+	{
+		const std::string& base_filename = entry.first;
+
+		if (!strstr(base_filename.c_str(), "_mip"))


degasus · 2015-03-01T23:20:02Z

mimimi: It's more about png decoding than disk loading, at least here for me on my hard drive.

Source/Core/VideoCommon/HiresTextures.h

@@ -40,6 +42,9 @@ class HiresTexture
 	std::vector<Level> m_levels;

 private:
+	static HiresTexture* Load(std::string base_filename, u32 width, u32 height);


BigheadSMZ · 2015-03-02T14:50:58Z

Sorry for the delay, didn't notice this until now. Gave it run and it seems to work as intended! This route turned out to be a lot better than I thought it would, my impression was the game would need to be paused or no textures would show up until after preloaded. At first I wasn't sure if it was working because I immediately seen custom textures, then a little message popped up "Custom Textures loaded, 3338.2 MB in 22.5 s". Ran around the known trouble spots and to my surprise not a single hiccup.

Linktothepast · 2015-03-02T16:46:16Z

It is working pretty well, but i did notice a couple of slowdowns here and there. Unfortunately it is also a ram killer, it makes pngs about 10 times bigger in memory than what they are. Tried two packs i have and 50 mb pngs became about 500 mb in memory and 70 mb about 800 mb. Big packs like the rogue squadron one which is 500mb will probably bring even a 8gb system to it's limits.
Generally though i like it and it works pretty much as intended providing you have a lot of spare ram at your disposal.

degasus · 2015-03-02T21:17:38Z

@lioncash fixed

BigheadSMZ · 2015-03-03T05:52:33Z

I can see that being a concern. What would happen if the limit is breached, as in it tries to preload 8GB of textures and only 4GB is available? And would dds files be smaller in RAM than png and be worth using? Or how about even jpg if it could be added (if it isn't already)?

degasus · 2015-03-03T06:46:20Z

@BigheadSMZ dds can be smaller, but in the current implementation, they aren't. And if you're out of ram, dolphin may just crash if you're lucky, else the system will freeze.

degasus · 2015-03-03T08:58:47Z

So any opinions about merging?

BigheadSMZ · 2015-03-03T11:59:07Z

While I think this is an awesome feature especially since I can probably take full advantage of it without worry, I don't know if it is safe in the hands of the masses if it has a chance to freeze their system. I imagine it will create panic when someone's 2006 laptop with 2GB RAM deadlocks using Dolphin and texture packs and they don't know why. I'm fine either way. But if there is no way to prevent a random meltdown then maybe it should have a "may cause system instability" warning.

JMC47 · 2015-03-03T12:00:00Z

This seems like an awesome feature, but can't we at least warn users with less than 8GB of ram not to use it? Or disable it altogether?

neobrain · 2015-03-03T12:14:22Z

This patch needs a proper host memory manager to be sensibly merged. Enabling options should not make it possible to destroy a stable emulation experience.

EgoBizarro · 2015-03-03T20:25:51Z

This should only be an issue for big texture packs. Most packs aren't that big.
I, personally, like this solution much better than the 'cache loaded textures' one. With the latter there would still be stuttering every time you play (though, obviously, it's still better than the current state); you'd never be able to have a smooth play. So I hope this one makes it into master sooner or later.

Source/Core/DolphinWX/VideoConfigDiag.cpp

@@ -138,6 +138,7 @@ static wxString xfb_virtual_desc = wxTRANSLATE("Emulate XFBs using GPU texture o
 static wxString xfb_real_desc = wxTRANSLATE("Emulate XFBs accurately.\nSlows down emulation a lot and prohibits high-resolution rendering but is necessary to emulate a number of games properly.\n\nIf unsure, check virtual XFB emulation instead.");
 static wxString dump_textures_desc = wxTRANSLATE("Dump decoded game textures to User/Dump/Textures/<game_id>/.\n\nIf unsure, leave this unchecked.");
 static wxString load_hires_textures_desc = wxTRANSLATE("Load custom textures from User/Load/Textures/<game_id>/.\n\nIf unsure, leave this unchecked.");
+static wxString cache_hires_textures_desc = wxTRANSLATE("Load custom textures on startup.\nThis may require much of RAM but fixes some stutter.\n\nIf unsure, leave this unchecked.");


neobrain · 2015-03-03T22:04:58Z

@StripTheSoulwhostolemydamnname "What does it matter, it only happens when using X anyway" is a terrible way to make decisions affecting software quality.

JMC47 · 2015-03-04T00:53:12Z

Maybe disable this option on configurations without much ram for now? or just make sure it doesn't max out ram and tell it to stop/abort/notload any more textures?

BigheadSMZ · 2015-03-05T23:39:04Z

Some texture packs are not very big, 50-100 small textures will not take much RAM so disabling the option based on a set value wouldn't really be fair to those who could technically use one pack but not another. Maybe a dynamic limit could be imposed like....

check amount of RAM in system
calculate size of pack in RAM before actually loading into RAM
if calculated size greater than available RAM (+ some min overhead?) then do not preload
show an OSD message "preload failed because not enough RAM" so its known why

I imagine this would also have a problem if the limit is nearly reached, loading Firefox, WinAMP, etc, could deliver a final blow that Dolphin wouldn't have reached on its own if only like 3MB was free after preloading. So maybe leave some overhead, fail to preload if at least (example random number: 512MB) is not calculated to be free. Such as...

8GB RAM in system
Calculated pack size is 6GB in RAM
1.5GB consumed by OS + other processes
If 7.5GB- total consumed, 512MB still free, preload
If 7.6GB+ was consumed, 512MB exceeded, fail preload

This way the OS still has at least 512MB free (or whatever value works best) to manage so hopefully the limit won't be breached. I don't know many details of how things work on the software level, so I don't know if this could be an actual approach or if I'm just spouting nonsense. Maybe other options could be to remove some textures from RAM or not load some on start, but that kind of defeats the purpose.

degasus · 2015-03-06T09:06:35Z

Checking for the size of all textures without loading them still takes quite a long time. So I suggest to just try to load them and when you hit the ram limitation, stop it.

atm I'm thinking to just use 50% or the global system memory. Free memory isn't good as it may change while playing. So we still can get slowdown because of out-of-ram, but no system freezes any more.

degasus · 2015-03-14T09:37:16Z

Updated to fix the remaining GUI issues and when to restart the prefetcher thread.
Now it's only missing when (or if) to interrupt the prefetcher on out-of-memory issues. But tbh, I have no clue how to do this on all plattforms :/

There is no nice way to correctly "detect" the "used" memory, so we just say we're fine to use 50% of the physical memory for custom textures. This will fix out-of-memory crashes, but we still might run into swapping issues.

degasus · 2015-05-16T13:48:47Z

The last commit now aborts the loading if not enough memory is available. It's only tested on linux, so may anyone please check it on osx and windows?

It shall abort if the physical memory is smaller than 2 times the decoded texture pack.

Linktothepast · 2015-05-16T17:46:14Z

Here in windows it loaded textures up to half the physical memory in RAM (2 gbytes were used by dolphin). It didn't abort loading them altogether though.

degasus · 2015-05-20T09:22:29Z

In my opinion, this PR is ready. Are there still any suggestions or questions?

Linktothepast · 2015-05-20T12:34:57Z

Testing wise no problems here.

JMC47 · 2015-05-20T19:36:10Z

No issues here using the Rogue Squadron tex pack.

CustomTexture: prefetch all available textures

lioncash reviewed Mar 1, 2015
View reviewed changes

Source/Core/VideoCommon/HiresTextures.cpp

{

const std::string& base_filename = entry.first;

if (!strstr(base_filename.c_str(), "_mip"))

This comment was marked as off-topic.

Sign in to view

lioncash reviewed Mar 1, 2015
View reviewed changes

Source/Core/VideoCommon/HiresTextures.h

@@ -40,6 +42,9 @@ class HiresTexture

std::vector<Level> m_levels;

private:

static HiresTexture* Load(std::string base_filename, u32 width, u32 height);

This comment was marked as off-topic.

Sign in to view

degasus force-pushed the prefetch_tex branch from 0ef66c3 to a6c53fd Compare March 2, 2015 21:17

lioncash reviewed Mar 3, 2015
View reviewed changes

degasus force-pushed the prefetch_tex branch 2 times, most recently from e499ea3 to d1190c4 Compare March 14, 2015 09:35

degasus force-pushed the prefetch_tex branch from d1190c4 to 09ca504 Compare March 29, 2015 08:46

degasus force-pushed the prefetch_tex branch 2 times, most recently from 8eb8adc to 82ff988 Compare April 7, 2015 09:56

CustomTexture: prefetch all available textures

242f7d9

degasus force-pushed the prefetch_tex branch 6 times, most recently from 6e94ccc to 562b9d4 Compare May 16, 2015 13:38

VideoCommon: Abort texture prefetching on low RAM

562b9d4

There is no nice way to correctly "detect" the "used" memory, so we just say we're fine to use 50% of the physical memory for custom textures. This will fix out-of-memory crashes, but we still might run into swapping issues.

degasus added a commit that referenced this pull request May 26, 2015

Merge pull request #2162 from degasus/prefetch_tex

a6412fb

CustomTexture: prefetch all available textures

degasus merged commit a6412fb into dolphin-emu:master May 26, 2015

degasus deleted the prefetch_tex branch August 9, 2015 08:59

jeremybris mentioned this pull request Feb 15, 2021

Dumping Textures from RSX Debugger RPCS3/rpcs3#1659

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CustomTexture: prefetch all available textures #2162

CustomTexture: prefetch all available textures #2162

degasus commented Mar 1, 2015

degasus commented Mar 1, 2015

magumagu commented Mar 1, 2015

degasus commented Mar 1, 2015

magumagu commented Mar 1, 2015

magumagu commented Mar 1, 2015

mimimi085181 commented Mar 1, 2015

This comment was marked as off-topic.

degasus commented Mar 1, 2015

This comment was marked as off-topic.

BigheadSMZ commented Mar 2, 2015

Linktothepast commented Mar 2, 2015

degasus commented Mar 2, 2015

BigheadSMZ commented Mar 3, 2015

degasus commented Mar 3, 2015

degasus commented Mar 3, 2015

BigheadSMZ commented Mar 3, 2015

JMC47 commented Mar 3, 2015

neobrain commented Mar 3, 2015

EgoBizarro commented Mar 3, 2015

This comment was marked as off-topic.

This comment was marked as off-topic.

neobrain commented Mar 3, 2015

JMC47 commented Mar 4, 2015

BigheadSMZ commented Mar 5, 2015

degasus commented Mar 6, 2015

degasus commented Mar 14, 2015

degasus commented May 16, 2015

Linktothepast commented May 16, 2015

degasus commented May 20, 2015

Linktothepast commented May 20, 2015

JMC47 commented May 20, 2015

CustomTexture: prefetch all available textures #2162

CustomTexture: prefetch all available textures #2162

Conversation

degasus commented Mar 1, 2015

degasus commented Mar 1, 2015

magumagu commented Mar 1, 2015

degasus commented Mar 1, 2015

magumagu commented Mar 1, 2015

magumagu commented Mar 1, 2015

mimimi085181 commented Mar 1, 2015

This comment was marked as off-topic.

degasus commented Mar 1, 2015

This comment was marked as off-topic.

BigheadSMZ commented Mar 2, 2015

Linktothepast commented Mar 2, 2015

degasus commented Mar 2, 2015

BigheadSMZ commented Mar 3, 2015

degasus commented Mar 3, 2015

degasus commented Mar 3, 2015

BigheadSMZ commented Mar 3, 2015

JMC47 commented Mar 3, 2015

neobrain commented Mar 3, 2015

EgoBizarro commented Mar 3, 2015

This comment was marked as off-topic.

This comment was marked as off-topic.

neobrain commented Mar 3, 2015

JMC47 commented Mar 4, 2015

BigheadSMZ commented Mar 5, 2015

degasus commented Mar 6, 2015

degasus commented Mar 14, 2015

degasus commented May 16, 2015

Linktothepast commented May 16, 2015

degasus commented May 20, 2015

Linktothepast commented May 20, 2015

JMC47 commented May 20, 2015