Please try using magumagu's stuff for this.

https://github.com/dolphin-emu/dolphin/pull/259

You can also build said build from his repository if you don't know, but I assume you do. I'm not good at github, so I don't know if what I said makes sense.

Just built and ran commit bf3315d from magumagu's "dtk-rewrite" branch, and unfortunately the gradual background music desynchronization is still present.

Well, if you could record some audio dumps in HLE and LLE and maybe a video showing the issue, then I could cc maybe delroth and magumagu as they've been working on audio.

Settings to try to see if there's any effect:

• LLE audio
• setting the frame limiter to half speed (30fps)
• disable dual-core mode and idle skipping
• record your input as a movie (Emulation > Start Recording)

HIGHLY recommend disabling dualcore; it can allow audio to run full speed while the game lags.

Here's a Dropbox link containing:

• A screencast (3m30s) of a level playthough with HLE audio and single-core
• The audiodump of that playthrough
• A dump of the input of a second playthrough with LLE and at half speed (30fps)

The half-speed playthrough with LLE exhibited the exact same lag accumulation as the full-speed HLE playthrough.

https://dl.dropboxusercontent.com/u/3863316/taiko_playthrough.tar.gz - 284MB

Did you try single core as well though? If you tried single core and it didn't fix it, I'll move this to accepted.

Yes, dual core mode and idle skipping were disabled.

It sounds like there's some issue with interrupt timing here (how many emulated cycles it takes for an interrupt to trigger). Unfortunately, I don't know which one it is. That said, there are a limited number of calls to CoreTiming::ScheduleEvent (and its threadsafe variants), and pretty much everything time-related depends on those calls, so it's probably not too complicated to figure out for someone with a copy of the game.

Random thought: Dolphin currently assumes NTSC is 60fps, and the amount of desync being reported here is the same order of magnitude as the difference between 60fps and 59.94fps.

I'm a programmer but am unfamiliar with the Dolphin codebase. Is there anything I can do to help? Alternatively, is there a bug bounty or sponsorship process?

Well, first off, might as well clear up the suspicion about the field rate. https://github.com/magumagu/dolphin/tree/nstc-field-rate is a port of the old fix.

(And BTW, the old repo is still available at http://code.google.com/p/dolphin-emu/source/checkout . It's just hidden because it's not usually useful.)

Problem still observed in the nstc-field-rate branch (commit eec7a7a). I also experimented by slowing down the framerate with the expression (60.0f / 6.0f) instead of (60.0f / 1.001f). With this new constant, the framerate overlay displayed 10 fps, and most the animations were 1/6 speed. However, all game music was full speed, and the "beat indicators" still scrolled by at full speed and in sync with the music, except for the typical gradual accumulation of lag.

Okay, thanks for testing. Hmm, so it isn't sensitive to the framerate... that's good to know.

If you're willing, some more things to test; first, check if messing with CPU_CORE_CLOCK (PreInit() in SystemTimers.cpp) has any effect.

If that has no effect, all the relevant code must call either CoreTiming::GetTicks()or CoreTiming::ScheduleEvent() (or the related ScheduleEvent_Threadsafe and ScheduleEvent_Threadsafe_Immediate); those are the only ways for code to be sensitive to how much emulated time has passed. I really have no good guess for what exactly the game is sensitive to, though. My first idea: AudioDMACallback in SystemTimers.cpp . (Try messing with the value of the variable "period".)

Sorry I don't have any better ideas for diagnosing this.

Setting CPU_CORE_CLOCK to 730500000 (730.5 MHz) does the trick. I experimented with a few different values. 731 MHz is definitely too fast.

How much certainty is there regarding the correctness of the current 729 MHz value?

The game is sensitive to the exact value of CPU_CORE_CLOCK? That's... a bit of a surprise. That does give a starting point, though: very few things in Dolphin are actually sensitive to the exact value of CPU_CORE_CLOCK. The most obvious is mftb and friends, which actually count clock cycles. Please check if changing the value of TIMER_RATIO in SystemTimers.cpp has a similar effect, just to make sure there isn't something else strange going on.

As for the clock rate, it's would be quite surprising if it were wrong; among other things, libogc agrees with Dolphin, and 729 is divisible by 3.

I'll have to think a bit more about where to go from here.

Some discussion on IRC brought up the possibility that the audio is playing at the wrong speed, and the gameplay is at the correct speed.

Do you have a setup where you can record this game running on a console? If so, can you compare the audio to see if it's playing at the correct rate? If not, maybe you can compare to a YouTube video or something like that?

Changing TIMER_RATIO from 12 to 13 caused a similar (more drastic, of course) desynchronization of the audio track and beat markers.

More experimentation:
In contrast with setting CPU_CORE_CLOCK to 730.5 MHz in PreInit(), merely adjusting "period" in AudioDMACallback by substituting a clock rate of 730.5 MHz local to that function did not fix the problem.

Also, did I mention (or was it already clear) that the audio synchronization is reset each time the song/game level is restarted?

Question: What's the difference between "AIDSampleRate" and "AISSampleRate"? AISSampleRate is streaming, right? Would it be safe to assume that the music track is "streaming"? I wonder if the audio track may have been recorded at some non-standard rate.

(Yes, you mentioned the sync resetting before.)

AIS is "streaming", but it refers to a specific kind of streaming directly from the DVD drive to the audio output; as far as I know, it's never used in Wii games.

Looking into it a bit more, AudioDMACallback is probably not the right thing to adjust... not sure what is off the top of my head.

I don't have a physical console setup, but I do have an mp3 of one of the songs from the game. The start of the game music (left speaker) is lined up with the start of the mp3 (right speaker) at 30 seconds into the following recording:
https://dl.dropboxusercontent.com/u/3863316/master%2078b3eb1%20-%20pulse%20audio%20backend%20-%20HLE%20-%20no%20idle%20skipping%20-%20no%20dual%20core%20-%20framelimit%20auto.mp3

They gradually get out of sync. Here are some measured times:

Common Start: 0:30.14

End 1: 2:29.90 (dolphin emulator)
Duration 1: 119.76

End 2: 2:30.16 (actual mp3)
Duration 1: 120.02

Duration difference: 0.26
0.26 / 119.76 = 0.00217

Clock speeds (MHz):

Nominal: 729
Experimental: 730.5

Difference: 1.5
1.5 / 729 = 0.00206

Hmmm... not conclusive, but it does seem very unlikely the producers of the game would speed up the song by 0.2%, so probably the audio in Dolphin is playing too fast. That definitely narrows down what the issue could be... but I'm not sure what it is. I'll see what I can come up with.

Could you check if adjusting "period" in AudioDMACallback has any impact at all on the audio playback rate? It does in some games, but not in others, so I'd like to know what kind of audio system we're dealing with.

Adjusting "period" in AudioDMACallback by a factor of 2 introduced heavy static/crackling into the audio, but did not impact the rate of audio playback.

Here are some things that did not improve the audio sync (and had no perceptible effect whatsoever):

• Changing "g_AIDSampleRate" +/-0.2% everywhere in AudioInterface.cpp
• Chaning "IPC_HLE_PERIOD" +0.2% in Init() in SystemTimers.cpp
• Changing "g_CPUCyclesPerSample" +0.2% in RegisterMMIO() in AudioInterface.cpp
• Changing the return value of GetAIPeriod() +/-0.2% in AudioInterface.cpp
• Changing "cyclesLate" +/-0.2% in Update() in AudioInterface.cpp

What's the best way for me to determine whether certain code paths are being executed (and how often), or to inspect a certain variable's value? Is there a logging framework? Is there a preferred IDE (or Eclipse support)?

So the audio playback isn't based on the DMA callback rate; good to know.

There is a logging framework; you can use WARN_LOG and friends in the code, and there's a log configuration window in the Dolphin GUI.

I use Visual Studio myself, but any IDE with C++ support should be usable for debugging.

Can you test the audio DMA callback thing again using DSPLLE? DSPHLE has its own independent timer which makes things confusing.

FYI, I just did some tests with the first three TAIKO games (R2JJAF, S2TJAF, S3TJAF). The music loses synchronization with the gameplay in the same manner as "TAIKO WII4"/STJJAF (which I have been testing this whole time), but for for the other three games, the CPU_CORE_CLOCK hack has no effect, even after cranking it up to 900 MHz. I verified again that 900MHz CPU_CORE_CLOCK does still drastically affect the gameplay speed in STJJAF.

Huh, interesting.

Please take a look at my previous comment: "Can you test the audio DMA callback thing again using DSPLLE? DSPHLE has its own independent timer which makes things confusing."

OK, I've confirmed that multiplying the period by 401 then dividing by 400 (0.25% increase) perfectly synchronizes the sound in all four Taiko games (LLE only; this doesn't seem to help HLE).

That definitely narrows down the issue.

I just did a hardware test to double-check, and the DMA interrupt does in fact fire at the same rate as Dolphin, so that isn't actually the issue here.

I'm not sure what the issue could be at this point... maybe it's reading the audio DMA control registers and we're returning slightly incorrect values? (AUDIO_DMA_BLOCKS_LEFT and friends).

OK, here's my high-level understanding thus far:

Given that the audio is playing too fast, and assuming that the timing of the DMA interrupts is already correct, then too much audio data is being processed in the span between each DMA interrupt.

I'm eager to poke around in the code and do more experiments. So far I've read these two writeups:

• "The DSP for dummies": https://code.google.com/p/dolphin-emu/wiki/LLEInformationAndIdeas
• "Emulating the Gamecube audio processing in Dolphin": http://blog.lse.epita.fr/articles/38-emulating-the-gamecube-audio-processing-in-dolphin.html

Is there anything else I can read to understand the DMA/audio pipeline better?

Could you clarify a few things? Sorry if some of these questions are irrelevant, and please let me know if there'a better forum for them.

• What constitutes a "block" (as in AUDIO_DMA_BLOCKS_LEFT)? Do they relate to "Parameter Blocks"?
• Is there a 1:1 correspondence between Parameter Blocks and discrete sound effects played in the game? Or would a long-running musical track span multiple Parameter Blocks (perhaps the music track is partitioned into one PB every 5 milliseconds)?
• What's the relevance of 0x1000 on this line: https://github.com/dolphin-emu/dolphin/blob/3afa17f752a06c438d912c85519d3edb34023447/Source/Core/Core/HW/DSP.cpp#L411
• Is there an equivalent hack to modifying the "period" value inside AudioDMACallback() that would affect the HLE pipeline the same way as the LLE?
• What functions in the source code does audio data journey through on its way from the ROM to the speakers (in other words, how does the code map to Figures 3 and 6 in "Emulating the Gamecube audio processing in Dolphin")?
• Taiko song tracks are stored as .nub files, and apparently no one really understands that format (see [1], [2], [3]), but might I expect to be able to correlate data in those files to data in "ARAM"? Is there a way to dump ARAM?

[1] http://forum.wiibrew.org/read.php?21,41066,41066
[2] http://gbatemp.net/threads/taiko-drum-master.188772/
[3] http://forums.ffshrine.org/f63/wii-music-ripping-guide-tools-69234/

Reading the relevant bits of yagcd might be helpful... otherwise, I can't think of much else.

A "block" in the AUDIO_DMA_BLOCKS_LEFT is just eight 16-bit stereo samples; this is the output from the DSP, not the input.

0x1000 on DSP.cpp line 411 is the size of the MMIO address space dedicated to the DSP and related processing.

I forget exactly where it is (DSPHLE.cpp?), but DSPHLE basically only runs once every 3ms.

UpdateAudioDMA() pulls the data directly from the DSP's output and sends it straight to Dolphin's audio mixer code (AudioCommon/Mixer.cpp). You're probably not going to find anything useful on that end; the mixer state is basically independent from the actual emulator state (e.g. if you savestate+loadstate, the audio buffered there is lost.)

ARAM is referred to as EXRAM on the Wii. Dolphin's debugger mode should let you view/dump EXRAM; it's accessible from the CPU at address 0x90000000.

1. Restating an assumption: Calling the DSP::UpdateAudioDMA() function more frequently causes more audio data (song span) to be sent to the speakers.

   1a) Halving the period inside AudioDMACallback() results in a song that plays at double speed (but not an octave higher in pitch), the song finishes early, and the latter half of the drum notes scroll by in silence.

2. According to your last response, by the time AudioDMACallback() is executed, the audio data it deals with has already been processed by the DSP.

   3a) With a halved AudioDMACallback() period, given that the audio data from later-than-otherwise in the song is already available to be sent to the speakers, this implies that the DSP has consumed twice as much data from the EXRAM (ARAM). To paraphrase, the effects of altering the AudioDMACallback() period must reach all the way back in the audio pipeline to change the rate of feeding EXRAM data into the DSP.

   3b) By what mechanism does AudioDMACallback() affect the rate of these earlier pipeline stages? There's not a direct function call chain as far as I can tell, and I'm having difficulty making sense of the interrupts. I assume it's to do with GenerateDSPInterrupt(DSP::INT_AID), but I lose the trail of anything relevant after ProcessorInterface::SetInterrupt(ProcessorInterface::INT_CAUSE_DSP, true).

3. The function Do_ARAM_DMA() seemed like it was worth investigating with all of the mentions of game-specific constants. However, Do_ARAM_DMA() is only called a couple of times upon game initialization, and so must be independent of the AudioDMACallback() period.

   4a) What is the rationale and etymology behind the ForceExceptionCheck() function?

4. The value of g_audioDMA.AudioDMAControl.NumBlocks in the "mmio->Register(base | AUDIO_DMA_CONTROL_LEN" callback is always 12, as is g_audioDMA.BlocksLeft in the "mmio->Register(base | AUDIO_DMA_BLOCKS_LEFT" callback. What is the significance of 12 in that context? I don't suppose it's related to TIMER_RATIO?

Answers:

3b) Once the game requests an interrupt, the cpu core will service the interrupt at the next available opportunity. For performance reasons, the interrupt is checked at the end of the timeslice (MAX_SLICE_LENGTH) instead of at the end of the instruction. When the interrupt is checked, Dolphin executes the CheckExternalExceptions function. The CPU then executes the exception handler in the Wii's RAM.

4a) The ForceExceptionCheck shortcuts the CheckExternalExceptions function to start executing after the current JIT block as some games will timeout if the CheckExternalExceptions function is not called soon after the interrupt is requested by the game.

5. Not sure about this one, but I think the NumBlocks is controlled by the game. Typically, it will be equal to 12 on a Wii game and 8 on a GC game.

I do not know a lot about DSP, I think more simply.

In DSPAccelerator.cpp, 'if ((Address & ~1) == (EndAddress & ~1))' replace to 'if (Address > EndAddress)'.

Please try it. I hear music by this correction definitely.
I'll leave that to your judgment.
(I also have taiko series, and I'm Japanese. Of course, I know J-POP in Taiko series. Sorry my bad english.)

hk.konpie, would you please test if the patch here fixes the Taiko series audio going out of sync?

https://code.google.com/p/dolphin-emu/issues/detail?id=7280#c10

skidau, I tried it patch in new rev.
I do not give a close test(check accurate length of the music, etc), No problem both HLE&LLE at taiko series. Looks good to me.
(Probably 'Rhythm Heaven Fever' may be improved, too.)

Beat the Beat Rythm Paradise seems improved too, in the past it took me a lot of tries to beat the monkey watch minigame while i did it on my first try with this build.

Fix confirmed in 4.0-2836. There is no longer audio desynchronization in Taiko. Very nice!

Is it really fixed??. I still feel audio out of sync as the song goes.

From the begining to middle of a song it is tolerable,
(actually it is perfect in the beginning)
but at the end of a song, I
can clearly feel that audio sync is slightly behind. I've tried every single released version.. but no luck. Im so sorry to open an issue but I really really
want to Taiko no Tatsujin in perfect condition.. right now it is not playble for me.

Are you sure you're using the right version? Latest development builds are here: https://dolphin-emu.org/download/ .

Yes, I tried the latest version.. I even tried it on my mac..but it had the same sync problem.. I'm not a programer and have no idea about coding, so I can not help out solve the problem but I'm 100 sure there is still audio lag issue..as the original issue subject it gradually accumulates and gets worse at the end of a song..
I'm a real rhythm game gig(every konami game) so I might be picky but it is what it is...

I heard a few rumblings on the forums a while ago that it is only fixed in some of the games?

yh876, I'm wondering if you can measure how much lag there is and let us know? I have seen something like this, where the audio lags by a ratio of 1/1000 (every one thousand seconds, Dolphin adds one second of audio). As you can imagine, this is very subtle (I measured it by comparing a recording from my Wii vs an audio dump from Dolphin). Is that the sort of magnitude you are seeing (hearing)?

Yes the audio out-of-sync is very subtle(but huge to me). I think it is because the frame rate limit is 60Hz. My monitor refresh rate is 59.95Hz. Since the program refresh rate is slightly fast, 0.05hz only at the end of the song I can feel the delay.. Is there anyway for me to set the frame rate limit to exact 59.95Hz??

Not at this moment.

According to pauldacheez

"skid_au made a branch a few months ago that changed NTSC to 59.94 as an experiment, though the intended purpose back then was to reduce audio latency (spoiler: it helped, but IIRC it was only because AudioCommon back then had less latency when the game was running at less than 100% speed). It's no longer available (nor is the rest of our GCode git repo), but the change is brain-dead simple, so you may want to redo it and see if it does anything useful."

I thought you might help me.. is there anything I can do on my side? pauldacheez said the change NTSC refresh rate is fairly easy... if you can give a brief guideline I wanna give it a shot.

When I tried that experiment, it didn't seem to fix the problem for me, but you are welcome to try it for yourself. You can download it here:

https://dl.dolphin-emu.org/prs/pr-1927-dolphin-latest-x64.7z

You're right. It does not work.. thank for your help though.