I'm trying to make sense of the core operation. Again. But this time for real.

I got a zoo of threads happily mucking about and dealing your music, with rough edges in user-friendliness regarding the new effects, but basically working. This took quite some development, one might be tempted to say "evolution" (but I'm talking intelligent design, here, eh?), from the root of one big master thread handling all actions, relaying orders to everyone, centralized mixing and management. Now the mixer thread doesn't do mixing anymore. It's been reduced to asking the input channels to please mix into their output and then releasing those prepared output buffers to the wild (poking the output channels). It does do management, though. Idle things like reporting status to clients, for example. That should go.

The Jam Master main thread should focus on one thing only: Keeping things in sync. Or ... does it even need to be present for that? What about input channels perpetually striving to fill their respective output channels as long as those signal that they're got empty buffers in the chain? Well, is that feasible or just madness? I have one goal of maximal parallelization and decoupling, but also another of timely reaction to client requests. But, well, considering that the output buffer chains are not supposed to grow disproportionaly, the time delay for asking a channel to change its volume won't change much with either approach. Just what vanishes is a bit of predictability. Assuming the main thread handles things and receives two requests to change channel volume in one go, it will apply the volume to both threads and then ask them for data. The change will apply at the same time.

But: Is there actually a way to ensure that two requests occur in a common time slot? I got my randomness right there, so nothing lost if the Master does not synchronize. There might be two script actions triggered at a common time stamp. They can be put into the command chain at the same atomic operation (for some value of atomic). If other channels, subject to those actions, fill output buffers as soon as they could, those triggered actions might be evaluated with one mixing block difference. Is that a problem? Also, how does the envisioned self-triggering of scripted actions play at all with the current approach of checking for actions before ordering playback blocks?

What is the promise to the client about execution of requests? First one is that the action will be executed some time soon, give or take at least one mixer buffer, plus delay from output buffer chain. It's more complicated with timed script actions. There, the promise is that if you scripted for time x in channel y, the action will, hm, happen while passing time x in channel y, ideally. Now what does that mean? The old approach interpreted this a bit eagerly: Actions are executed before playing the buffer that contains x. That honors a rather strict interpretation: If there is a volume change for time x, the sample (yeah, I should call it "frame", I know) at time x should be affected. I might get away with saying that samples at most a mixer buffer away from x should be affected. But more than that is just gross. Oh, and while we're at it: In a far future, something like a volume change should really occur at the concerned sample, in the most extreme case, any input parameter change (think effects ... should call those "filters" again ... whatever) should be exactly timed by dividing the to-be-delivered buffer into chunks and fetching them separately. But: My main goal is continuous playback and control for live operation, not coding an overly complex DAW. There needs to be some independent stoic processing into non-miniscule buffers to have any sense in the whole multithreaded mess. There's a reason something like Ardour wasn't designed with many threads from the beginning.

OK, back up from the madness. Think in granularity of mixer buffers. When the channels start to push their data asynchronously (they already mix into the same output in unspecified order, but that's just a bit of numeric detail, I'm not going into bit-identical results here), time x in channel y does not have a fixed relation to time x in channel z. That is bad. No, I'll stop going down that path.

So, the Master keeps the time. Channels are synchronized when being asked to mix in. What I still want to avoid is the Master having to block to read some channel state. Also, it should care as less as possible about channel details. I see one chance to move the script collection into the channel threads: Just parallelize the loop looking for scripts! I'd have two rounds, then ... and it only is of benefit if there is indeed a mass of script actions to check ... or if I'd be happy about accessing the script data structures always from the same thread until pushed to the main actions (the jury is still out on something like the main actions acually existing ... or rather action queues for the individual channels, with a priority spot for the mix-in action).

Currently, there is an optimization of keeping a list of channels that actually have scripts pending. Well, there could be a semaphore representing the scripter count. If there is at least one, ask all channels about scripts. That would be a simple optimization for the case of having no script at all. Another would be to cache the next time a script action is needed; but that time is separate for each channel, so something the Master won't care about.

Something the Master does care about is the question whether a thread is availble for any action at all. Currently, there is channel state indicating that, namely the "working" flag. More elegant would be a list with channels that are available for action, and individual channels taking care of dropping/adding themselves from that list. But how is that flag currently synchronized? There is the big playlock, with the main purpose of throttling I/O-heavy background work that might interfere with mixing (based on experience ...). It's ugly, but it seems to be necessary, unless the background workers throttle themselves unconditionally --- another issue for another time. For now, there is this lock that is held by the Master while calling inchannel::on_air() which checks the "working" flag). Where does the writing to the flag happen? Ah, yes, in the Master thread. It sends the channel action and sets the flag, also unsets it on notification that the background task is finished.

So, there is no actual synchronization needed: The Master manages the whole deal. Now, what I am pondering here is getting away with that brokering.

Interlude, a corner case: What about scripting a channel to stop at time x? Do I want to stop it _after_ playing the buffer containing x or before that? That fully depends on the implementation and I cannot off-hand decide on what is correct. Well ... actually, I can: Reducing the ordered block to only reach to x and have the channel pause after this block. The pausing after playback would create a precedence for having actions before and after playback; though, after _is_ before. Keep that in mind.

And don't forget that end scripts being added by channels after playback. It has to be after playback because one does not necessarily know in advance where the end is.

Back on track: Can I remove that brokering by the Master? There are separate action queues for each channel already, only they are moderated by the Master. Couldn't the clients directly access those? One would need to ensure that they don't hog the channel while the Master wants its data. The playback action rules. Actually, it might really not be part of the queue at all but communicated on the side, so that regardless of what action wakes the channel worker, a pending playback request is handled first. This still leaves open the possible issue of an endless amount of clients hogging the communication path; the Master could be blocked from sending its own wakeup call to the channel worker. But if that same amount of clients would all talk directly to the Master, as it is now, would that really be better? I think I'm entering pathologic pondering here. Normally, clients wait for a response before triggering another action, and if not, I can limit the number of queued actions easily, as I can limit the number of clients. This could be a protection from some DOS attack by coworkers who share one dermixd instance ... in a world where they might be allowed read-only access to the workings (auth and priviledges are post-2.0).

Eh, where was I? Clients submit channel actions directly to channels. Heck, they could even directly ask the mixer worker thread (not the Master, the slave) to create a new channel, or even delete one (outside playlock ... or other measures) --- but let's let that stuff be brokered by the Master for now. Talk about channel actions. Loading of tracks, seeking, managing effects, that sort of thing. Stuff that changes the on_air() state. When actual stopping of a channel because reaching end is handled in the channel worker, it is no biggie if the Master asks a non-on_air() channel to give some data and it reports a shortage. Heck! It doesn't even report anything anymore! That is a thing of the past. The Master could fail to ask a channel that just now happens to be going on_air(). But, well, it will get in next time.

And what about the Master not caring _at_ _all_ how many channels are playing? What about simply posting one big semaphore and having the channels run with it? It's only the common timing that is needed. Can I ensure that everyone gets the global signal once (re-posts after waiting)? This sounds like a standard problem with a standard solution from the book. Hm, but this won't work when the Master doesn't know how long to wait for the inputs to finish. It needs to know how many are ready to work and those need to get signalled specifically.

Ok, then, I do need either that "working" flag or a list of working channels that channels get pushed in/out of. If actions are not brokered, there needs to be synchronization. I do want to minimize the locks the Master has to take during its usual routine. One lock per channel does not look nice. One lock for the active channel list would be an idea (or some read/write barrier thing, perhaps one common one protecting all the channels' state variables), but how does that look for the individual channel? It wants to get busy, locks the state to set itself to "working", or alternatively remove itself from worker list, just while the Master has a hold on the data. To avoid instant deadlock, the Master must release the state lock after sending the playback orders and remembering how many answers to receive. Then, the channel wanting to get background-busy must always check if it got a playback order and then at least signal the Master that it's done (even if it did nothing to fulfill the order). 

This sounds like moving the playback order out of the ordinary work queue. There needs to be a special place for it. That matches up with my intend to prioritize it anyway.

So, are we settled? I intend to move the action brokering to the client handlers that issue them. Actions have flags noting who to send them to, the only actions being tended to by the Master concern overall operation: sleep and shutdown; or changing of core parameters like mixer buffer size. Everything that absolutely needs to be done to everything at once. Giving total status info (fullstat, if that may stay) does not belong into that category: this is something tended to by the individual channel threads in turn, perhaps with some provision to sort it later on. Or ... not? Do I want to concern the output channels with client communication? Perhaps this is still a job for the mixer worker thread, though there is a certain charme in the scatter/gather approach to fetch status info. But then, it's by no way computationally intensive to get that info, the only issue being the need to synchronize. I guess I don't gain anything in occupying n threads with communication overhead instead of having one thread grab the locks in turn and collect the data (with or without a separate state for the mixer worker, that separate state would need some locking at some point anyway). Individual status info should be directed to the input channels, no issue. But what with the outputs?

The output threads are mostly waiting for I/O (well, the latter part). They are not to be disturbed there. So I guess, it's still up to the mixer worker to respond to questions regarding the output channels. The facility to safely fetch status info by mixer worker should be common between the two channel types anyway. Here, I am closing the circle back to my earlier musings about the input state. It's not input state or output state, it's channel state.

So, let's start somewhere. I could side-step the hard part by starting with the fullstat info collecting. Where is it currently? Ah, yes, it's in the Master. This is a good start to move side-effect-free action to the mixer worker/messenger thread. Heh, looking at the header for it's class, I see that this was my idea from the beginning;-)

But this really needs some basic change: The mixer worker is currently just triggering the defsetup on startup and then stupidly watching its message semaphore, just to print out errors. Seems like that should be copied to a pure error sink thread and the current one extended to be a true messenger.

The error sink utilizes a comm_data instance that is used in promiscuous mode. Except, there is no such mode, it is just used as if this would be accepted practice. It's being raped, family-style. I just hand over the sink comm_data along with actions that might not have an own backchannel. Posting messages to comm_data needs protection. Eh, yes the mutex I put already in there. Is that used consistently? I see the mixer_sink (renamed from mixer_worker) locking on comdat ... is anyone else doing this?

Yes, the input channel worker is. Output worker is not yet really there yet ... mixer doesn't really qualify ... or ... hm, scripts may point back to the sink. I guess it should also grab the lock, especially since more processing will go concurrent. It shall be noted though that there is an exception for the client action handlers: They issue a command and wait, after the notification there is noone else messing with the comm_data. This is only different for the sink that catches messages without requests... eh ... what about watching? Right, that is at the socket writer level. Heck, why is there another level? More parallelism. It's a mess, alright. Get over it.

I have everyone locking on comdat, and also that one also functioning as mutex if there is real concurrency expected.

Now thinking about separating out the mixer actions, and enabling parallelism in the management while at that (so that I can move the functionality piecewise, and also could easily add a pool of threads doing the work, if that need arises). The channel management quickly raises questions: I see a difference between actions that disturb current operation (playback on both sides), like removing a channel, and actions that do not affect anyone per se, like adding a channel before binding it to anyone.

Apart from the issue of changing IDs (which still sucks ... just switching to a map instead?), a channel doesn't matter to other channels as long as it has no connections to them. One could optimize the process of adding/removing channels. Thinking of the linking: Why does an output channel actually have to know which input channels feed it? That's only for the benefit of the client, making sense of the connections! The output channel itself does not care at all. Hence, that redundant info should be removed and reconstructed when preparing the status messages to clients. Only reason for the double entry is easy detaching of outputs. Think about that.

So I can remove input channels at will while they are not in the critical mixing stage (playlock is there for that, let's keep it ... or, well, just read-blocking the rack does the trick, too). Only connections I have to worry about are the followers. I get the impression that followers aren't really handled in channel removal yet. Do I smell a segfault there? Yes! I can crash my daemon as predicted! Result! Easily fixed, though.

Back to the timing of scripts. Scripts being triggered at end of a track and at beginning of a follower necessarily come late. In the case of the starting follower, well, consider it like this: It officially starts on the next round, so its script actions are also executed there. Big deal. Normal script operation is to anticipate positions. But think about that: A track can end prematurely. It is only a guess that it will be able to fill the mixer buffer. Come to think of it, it is actually more correct to reformulate the promise about script actions:

	script n t command

This means that when channel n passes time t, command will be triggered. This is different from "command will be executed so that it affects samples at least beginning at t". The definition here is simpler and also possible to consistently implement. I'll change to that! Input channels will push scripted actions as they pass the corresponding times, the actions being scheduled to be executed before the next block. That makes sense, dammit! You just have to be aware of

	script n 0 command

being executed after passing position 0. That means, if you want to start some tracks at the same time, you better use an unrelated channel as timer:

	script 42 0 start 0
	script 42 0 start 1
	script 42 0 start 2
	start 42

That ensures that the first three channels start playback at the same mixer cycle, and they should stay in sync. Wondering, how would looping a track (not gaplessly) work?

	load 0 some_file
	nscript 0 10 -2 seek 0 0
	nscript 0 10 -1 start 0

That should do the trick to loop the track 10 times. Testing ... damn, I got a bug:

showscript 0
[showscript] +error: Backtrace follows, beginning at lowest level.
inchannel: Input channel is busy -- it will not accept any action until the mixer got the notification that the work is done. (2)
[showscript] -end

Well, exactly that stuff has to be fixed anyway.

Ah, yes. Seeks need notification that they're finished. Who gets the notification for the scripted seek? Sounds like work for the mixer worker ... or ... why doesn't the channel notify itself? It could at least push the notification action itself to a mixer action queue ... no real need to bother the mixer_worker with this, eh? Actually, not even the Master thread should be bothered, once I got channel state safely synchronized.

DANGER! Script actions retain the original socket_writer pointer ... those get exorcised from watchers when disconnecting a client, but what with scripted actions? I bet that's not in there. Hah! No, it is. Phew. What I should change is, instead of setting it to NULL, making it point to the mixer sink instead. Said sink needs to create a socket_writer for STDERR, then. No biggie. But back on track now. Get that notification out.

I realize that since there is a separate input worker thread now, this notification loop via the client thread is superfluous. The whole business is stupid and wasteful now. The channel worker just should update its state itself. Until then, it should at least push the notification to the Master itself.

The logic is rather fragile now: One needs to make sure that the mixer notificaton arrives before the next action from the client. This is ensured if the mixer notification is pushed before notifying the client thread. But this bites me with the attmept to script a loop: The start command silently fails because the load is not done yet. I need to delay that.

Hey, I got a drum machine!

load 0 kick.wav
load 1 snare.wav
load 2 hihat.wav
load 3 hihat.wav
inload 4 dummy trigger

volume 2 0.3
volume 3 0.2

nscript 0 -1 0.5 start 1
nscript 0 -1 0.75 pause 0
nscript 0 -1 0.75 seek 0 0
nscript 1 -1 0.5 start 0
nscript 1 -1 0.75 pause 1
nscript 1 -1 0.75 seek 1 0

nscript 2 -1 0.25 start 3
nscript 2 -1 0.375 pause 2
nscript 2 -1 0.375 seek 2 0
nscript 3 -1 0.25 start 2
nscript 3 -1 0.375 pause 3
nscript 3 -1 0.375 seek 3 0

script 4 0 start 0
script 4 0 start 2

start 4

I still get the error about busy channels when pasting this all in one go, but that will be settled. Meanwhile I bathe in the delight of having DerMixD actually playing samples from RAM like a tracker/drum machine, due to the generous input prebuffer.

Yeah, the notification is not right yet, and even if it where, there'd still be some gambling with the above script: just assuming that the seek-back is quick enough so that the alternating start command is effective. I need to ensure that ... scripts should behave as if entered by the client, at least a load should complete to have following script commands effective. But: The load needs unspecified time. Perhaps I need a nother special time definition: "just after a load" or "just after channel becomes ready again", to cover seeking. That would still not make the above script safe, though. Worst-case, the start action should wait till the channel is there (would still be rather bad for a drum machine to produce such a hickup). But since my main intent is reliability, keeping going with a hickup is better than not keeping going. Yeah, what about not giving errors because of working status of channel and instead keep the action in the queue and let the client wait? Wouldn't that make more sense anyway? Yes, it would.

This error message is a cop-out anyway. What is a client expected to do? It will try again. Waste of time, dermixd could do that right away. The worst-case now would be some really long-running background op

But first, fixing that bug. After two days of banging my head against this (yeah, I _do_ have something better to do. Well, not better, but more important), I finally found that mixer::process_actions did always clear the main action list, even when called for script actions. I discovered before that the notifications get swallowed sometimes. Semaphore indicated work, but there was no work there. This is explained nicely now. Seems like, again, I got two bugs relating to this. The other one was treatment of the action list semaphore: Both zeroing and posting should rather be inside the locked section. Otherwise a signal could be lost. Not sure if this particular bug fired in this case, but it eventually would have.

Damn. It was a left-over typo from restructuring the mixer! All the time and lost sleep; time I ripped from other projects that are more pressing than this pet of mine! Parallel programming really tests your frustration tolerance.

Anyhow the poor-man's drum machine works now. Hihat doesn't stay in sync too long, as is not surprising, but the basic idea works. For proper drumming, I'd create a tracker channel ... or, more like a sampling input, one channel for a track. The one feature the sampling input would have is to integrate the repetition to have proper timing. Hm, or would this gimmick just be a property of the normal channel? I'm getting ahead of myself again. This is not core business.
