serialosc (non-zeroconf) multi-app browser techniques

  • DISCLAIMER: this is not for beginners, and at present doesn't do anything interesting — simply a proof of concept

    following from the discussion in @myr's monome_home topic, there was discussion of a p2p style monome splitting / panning / routing application (like pages) that works without zeroconf:

    here's what i cooked up in a couple hours since lunch. it really just suggests one technique of implementation.

    at present, all it does is allow you to connect 1 monome and route the presses/leds to/from any of the currently open apps. this is handled directly over UDP (host to app) so it avoids the need to change serialosc itself, nor the need to use zeroconf. it requires a modified serialosc.maxpat (attached in the zip) which should go in the usual place.

    does this seem a reasonable / solid-enough approach to this issue? what limitations are we likely to run into?

    also please note - i'm not intending to create an entire full-featured routing app, but rather wanting to suggest a practical alternative to restructuring the serialosc protocol again, as well as avoiding the non-maintained and problem-causing zeroconf externals for Max.

  • platform lock-in. win/mac-only, and requires max. unless this is specifically only to address limitations within max itself (and using that environment with monomes), is there a more cross-platform way to do this?

    i always imagined something like a central text file that every app would check and write to on startup, to see what's running where. while this would avoid bonjour, a central config would require rewriting all apps, plus serialosc, to take advantage of it. not as simple as your intermediary patch.

    griddle already does something like this, but it still relies on zeroconf. we'd need artfwo or some other python coder to remove that dependency.

  • looks great! thanks for the work. will have a chance to work with it and start extending it later.

    agreed that avoiding changing serialosc and avoiding zeroconf are likely the best options. i'm happy to extend this and work on a solution in max, as it's the only environment i use. hopefully there are no issues with other environments.

    the only limitation i think i can see so far is when using multiple instances of the same app. using the prefix to distinguish app instances is the best way, rather than device id as i suggested in the other thread, but i think there would need to add a /broadcast/prefix message to serialosc.maxpat also. this way the manager app could append sequential numbers to apps with the same prefix, or allow the user to change the prefix entirely. i.e an 8x8 app with 8 faders normally prefixed 8faders could have two open instances renamed to 8fadersVol, 8fadersPan.

    the problem that would remain though would be preseting or saving the setup in the manager app, closing the monome apps, opening them up again and trying to reload the preset. the manager app has no way of telling which app was which as now they've got a random port and the original prefix for all app instances.

    at the minute the only way i can think of working it is having serialosc.maxpat store some information between open/close. before i'd suggested storing the device id, which would require a manager app to create virtual devices with inidividual device id's for apps to store and connect to. storing a prefix does seem better though.

    this does cause problems like where and how the prefix or app identifier would be stored though. at least i can't think of a way that would work with max runtime, it's pretty easy in max for live or full max though. i had an old serialosc patch that stored device id, prefix and rotation in a text file on hard drive with the app, but that still wouldn't solve the multiple instances of the same app issue.

    would it be a good idea to add an app close message to serialosc.maxpat. could use [closebang] to send a message to the manager app and refresh the app list.

    also quick suggestion, in serialosc.maxpat i think you should make the pattr object that is bound to device id be invisible to pattrstorage, set and freeze the attribute in inspector. that way it won't show up in pattrstorage devices as u8495723789 or similar.

    thanks again for the work on this.

  • maybe we'd need to allow the serialosc.maxpat's prefix to be changed from outside and store it in a pattr object. we could then encourage app developers to store the prefix in a pattrstorage outside serialosc.maxpat.

  • @ioflow the ideas in the max patch are platform independent. it just means any apps built in other environments would need to include the same things in their app. currently that's just listening to and sending a few extra messages over another udp port.

    the managing app being discussed may be built in max, i'm likely to have a go at it, but other developers could program a managing app using the ideas discussed here in any language. just as they've had to program their own versions of the ideas in serialosc.maxpat.

  • @ioflow
    myr is totally correct. i simply used max to illustrate the concept because i'm most adept at expressing an idea with it! all the manager is doing is monitoring udp-port12003 for new apps, and then sub-distributing messages to/from child apps on behalf of serialosc.

    i agree the multiple instances of the same app is an issue, which is perhaps most of an issue within m4l. is this is a primary use case? do many people run a number of the same app? i know that most max apps don't create exclusive send and receives, so won't work properly if there are copies open. using the "---" prefix in m4l has allowed duplicates and so presents an issue.

    is this a primary goal though? perhaps this issue could be dealt with at a later point, and leave it as a limitation.

    regarding which params to save, i'm pretty set in my belief that it should be handled by the host app. this could be within mlrv when using it in a direct connection, having a specific device recorded within a preset. or this could be the preset file connected to the managing app in question, where it would record which prefix is saved where.

    this does create an issue with loading a preset in the manager if the app corresponding to the saved prefix is not open. i don't really have a solution for that. how does pages handle the opening / closing of child apps?

    on the use of closebang to remove an object from the dropdown, it's definitely a possibility but i always had issues with closebang not working properly. perhaps that's only when you quit Max altogether rather than just closing the instance, i should investigate but don't really have the inclination.

    regarding any changes to serialosc.maxpat, please feel free to go for it, i'm not precious about keeping it the way it is, and it's definitely messier than when it started out, but i'd just like to keep it nice and compact so it's easy to understand.

    re: hidden from pattrstorage — will do! frustratingly, every time you type a change into a frozen pattr object it changes back to default params and has to be re-frozen... will do it with an @arg instead

  • ----------begin_max5_patcher----------

  • @karaokaze:
    yes - the accum was just a hack to throw this together. using the umenu index makes far more sense.

    the random-port-selection method could definitely be shifted above the reserved range. fundamentally i was anticipating that a full roll-out of this concept would almost duplicate the serialosc.maxpat into a 'subosc.maxpat' type bpatcher with all the bells and whistles, except the dropdown selects the app rather than the device.

    also - i imagine there should be some form of communication from the manager app to the serialosc.maxpat inside each controlled app in order to show what's connected and being controlled.

  • "how does pages handle the opening / closing of child apps?"

    it mostly relies on optimism and the power of positive thinking. it really just sets up its side of the bargain as i don't think there's much else that can be done currently? sadly others probably know more about how this translates to the real world than me. i'm guessing each serialosc app has to be reconnected each time.

  • @galapagoose / @Myr: good point on merely using max to present the steps an app would have to take; developing a tool for each os & env from there should be straightforward.

    on the multiple app use case: pages and griddle both make it possible to run 2+ apps on the same grid. i've run two instances of rove on a split 128, and there are other times when one might need to duplicate an app. example: a program that sends midi, but doesn't have a built-in method for sending more than one sequence or on more than one channel. you might load up enough instances to address a few different softsynths and drum machines.

  • my main use will likely be in max 4 live, and will involve several instances of the same apps open. luckily this is relatively easy to solve in max for live as we can save information on each monome app with the ableton live set, without any writing to text file or anything.

    it could involve making a m4l version of serialosc.maxpat. could add a pattr object storing the prefix and then set its "parameter_enable" attribute to 1. this would make it save and recall with the device in the live set. this could be done outside of serialosc.maxpat though, would just involve storing the prefix as an ableton live parameter at some point.

    as for achieving multiple app instances outside of m4l, maybe it will have to be something handled by the monome app individually. unless the app says it can be used with multiple instances then it's assumed it won't work and is a limitation.

    it could be achieved by saving the device prefix in presets within the monome app. this would require users to load the right preset for each app on screen, or a bit more programming in the monome app which would keep track of how many instances of itself there were and change the prefixes accordingly.

    or could make a kind of master app, a bit like ableton live, which would be the shell within which you open your monome apps like modules. this master app would keep track of which apps you have open and recall prefixes and presets for each individual monome app.

    looking forward to starting some patching when i get home.

  • @karaokaze
    defs agree that cross-platform dev is important, but perhaps in this community-based software development we should be spreading this kind of workload across multiple devs. especially with a topic like this, it seems there's a great deal of interest from devs working on all platforms and environments.

    regarding the ports (a great example of how simple communication goes a good deal of the way to working x-platform), i'm not certain how their handled but perhaps we just need to operate over 2 separate ports (ie. one for from-manager, and one for to-manager). does the conflict occur when creating multiple sends or receives?

    personally my whole idea behind this patch was as a simple and direct connection between manager and children — as such i'd push back against over-complicating the functionality and just focus on the main goals to start with. build it gradually, rather than defining some huge list of required functionality that makes the task 1000x more difficult (i know i'm guilty of that with my coding efforts)...

    would still like to hear from a few more folks about issues with the fundamental idea we're considering here....

  • it would be nice to have a non-max explanation of this as i neither have max nor am proficient in it. also i really don't understand what the patch does even after reading the thread.

  • i'm not certain i understand the patch. is the idea that concurrent applications can be changed via a 'core' selector on a single grid device? like pages but for max?

  • i appreciate the efforts here and i think there are some interesting ideas! unfortunately this doesn't solve the problem i was hoping to solve, which is simply "how can i load a virtual device into serialosc" or "how can i write my own serialosc and have it play nice with the real serialosc", with the true motivation being "how can we make virtual devices without zeroconf". maybe i'm missing something? i see two actual potential solutions to this:

    1) modify serialosc to allow virtual device registration -- there is some effort underway on this already. this will probably involve a tcp/ip connection from the app wanting to register a virtual device that is maintained with serialosc. when the connection breaks, the virtual device goes away. this is to avoid leaky virtual devices.

    2) iterate over a series of "serialosc ports": 12002 - 120XY. the size of the range is arbitrary, but the concept is that i could simulate serialosc messages with my own process on 12003. say i was writing the equivalent of a serialosc for midi-based grids (launchpads, apcs, livids, quneos, etc)--let's call it "midiosc". i'd want to be able to run serialosc on port 12002 and midiosc on port 12003 and have serialosc-enabled apps send discovery messages to both. i think this means a minor modification to serialosc.maxpat to iterate on a port range and the equivalent change in other environments.

    so while i think the efforts here are interesting/cool/should be explored, i don't think they're addressing the problem i'm interested in solving. that's just my take on it.

  • @phortran the way i was looking at it, but managed not to mention in any comments, it could be a solution to the virtual device registration problem and it's a start at implementing point 2 in your post.

    the talk in this thread has focused on registering applications, but i was seeing applications and virtual devices as the same thing. a virtual device being a special type of monome app which lives inside a device manager application and connects to one or more real devices. apps which are virtual devices could be distinguished by their prefix.

    i was imagining the code in client applications, in max's case serialosc.maxpat, could be adapted to display devices and virtual devices in its drop down menu, or in a separate menu. admittedly i haven't fully thought it through yet, and haven't had a chance to work on it. bit the bullet and got a second hand ipad and mira, multi-touch!

    if there needs to be more of a distinction between virtual devices and client applications then maybe the procedure could be doubled and we could have 12003 for registering virtual devices and 12004 for registering apps.

    there would be one procedure for client applications (serialosc.maxpat in max) which would connect to real and virtual devices and a slightly different procedure (a variant of serialosc.maxpat) used in a manager application or virtual device which creates a list of client applications for a virtual device to connect to.

  • ok, so thinking of apps in terms of "could be a virtual device app" makes more sense, just a semantic issue in my brain i suppose. thanks for clarifying that thought. at the same time what other type of app beyond a virtual device would make sense? clearly you wouldn't be connecting mlrv to polygome.

    i'm also somewhat averse to the complexity of the proposed management protocol. it seems like we already have a serialosc protocol that could be borrowed and everything is so much simpler at that point, right? this statement is purely in the context of virtual devices, not considering any other benefits that a manager app might have.

    in other words, why build a management protocol when we already have one -- serialosc.

    to be clear i'm not in any way against expanding the ideas of app/device routing but purely in the context of virtual devices i don't see what we're gaining vs. a simpler solution.

    edit: just one additional thought re: apps and virtual devices, it seems to me there is a clear distinction in that an app receives presses and sends led commands (over simplified but i think it's valid). virtual devices send presses and receive led commands. that's how my brain is compartmentalizing these things at least.

  • agreed there is a clear disctinction. apps receive presses, send leds; virtual devices receive leds, send presses.

    Apps receive from a Device(real or virtual), send to a Device(real or virtual);
    Virtual Devices receive from an App, send to a Real Device;
    Real Devices receive from an App or Virtual Device, send to an App or Virtual Device.

    i don't know if there'd be any use case for a Virtual Device to be able to send to and receive from another Virtual Device; like windows within windows. but i guess that would be something different, like a Page, which would be a collection of Virtual Devices laid out across one or more Real Devices. something which is handled by the Manager App. a Manager App being a group of Virtual Devices and Pages, which specify which Apps connect to which Virtual Device and which Virtual Device connects to which Real Devices. a Manager App would also act as an App Server registering any Apps and storing the port and prefix information for connecting to Virtual Devices.

    in regards to protocol, it seems sensible to say Virtual Devices should at least be the same as Real Devices. they should send and receive the same osc messages as described in the monome osc protocol documentation.

    any changes to Virtual Devices such as what Real Device and App they are connected to should be done by the Manager App, and i guess that is up to the developer of the Manager App. however, the creation of Virtual Devices and how they appear in SerialOSC Server or a pseudo SerialOSC Server should be defined. should we say any pseudo SerialOSC Server should have the same protocol as the Real Device SerialOSC Server? so it should respond to /list and /notify messages, and send /device, /add and /remove messages.

    if the ability to register Virtual Devices with SerialOSC Server is implemented then it will obviously remove the need to think about pseudo SerialOSC Servers. and it would make relatively no difference to an Apps protocol, i.e serialosc.maxpat, as Virtual Devices would now just be displayed with Real Devices from SerialOSC Server and we would just be working on how to get Apps to be able to register themselves with an App Server/Manager App.

    i think the work Galapagoose has done defining some extra protocols for how an App can register itself with a Manager App and receive messages to connect itself to a Virtual Device is a solid basis for working on a Manager App. if the work on modifying SerialOSC itself to register Virtual Devices doesn't go ahead then maybe Galapagoose's work here on App registration can also be used for Virtual Devices, creating a pseudo SerialOSC Server. as it is like the reserving port 12003 etc, point you made in the Monome Home thread.

    [edited: removed the ", Phortran" at the end as it read kind of ominously in my head, which was not how it was meant at all of course]

  • thanks myr, i think we're completely on the same page with this stuff. i do think an app router/manager protocol is a separate discussion (and i think i'm the one guilty of going off topic here, as this thread is that discussion).

    i probably would not retrofit pages with the management protocol unless it was really needed/desired by people as it seems to get along alright as it is -- it registers virtual devices and the apps map to the devices individually, instead of registering the apps themselves. i am sort of half working on a "new pages" and i'll definitely be watching for developments in this area in regards to that, as it might be a good candidate for housing some of that management functionality. i definitely think there are interesting ideas to explore and we'll see if any of this starts to catch on!

  • yeah, sorry for the rambling thoughts, kind of thinking it all through by writing it down :).

  • i would like to be of more help on the registering virtual devices with serialosc proper side of things. but with my limited knowledge and what sounds like a steep learning curve, programming tcp/ip connections etc, maybe i should stick with working on an App server / Manager App in max.

    also i'm wondering if with what i've read about tcp, and the lack of support for it in max, it will be possible to create virtual devices with max? would serialosc open up this tcp connection somehow so devices could be registered over osc or udp? if not then is the idea that there'll be an app you need to run which deals with creating your virtual devices? and we could then make another app that would allow you to page between these devices? or would it all have to be programmed in a more conventional language which can do things like tcp more easily?

  • well, the idea of a tcp connection may be short sighted in that regard if it locks out a lot of platforms. just doing a quick google it looks like mxj is a possibility in the max world. i'm not sure on pd / supercollider, processing should be fine, chuck might be an issue. if anyone has any insight into tcp/ip on these (or other) platforms i'd love to hear it!

    edit: i fully intend to go ahead and make the serialosc.maxpat mod to iterate over the ports and modify pages to include the serialosc discovery protocol as it would be great to avoid the zeroconf requirement in pages and it's something i can just do and put out there for feedback--a stopgap for now until we figure it all out. tried last night but it just wasn't happening, hopefully in the next few days though. props to galapagoose as well for not only posting the idea but including an implementation!

  • nerdiest thread ever! :)

  • just reading over my last few comments i realised i failed to mention the other good thing about Galapagoose's idea and implementation; it stops the need for virtual devices to be registered on SerialOSC Server or a pseudo SerialOSC Server. it does this by allowing the manager apps (apps that create virtual devices) to send information straight to monome apps (clients), and make them connect to a virtual device. therefore virtual devices don't need to be shown in a monome app's (client/serialosc.maxpat) drop-down menu. you use the manager app to connect virtual devices to monome apps, rather than the current method of connecting monome apps to virtual devices, which are in a manager app.

  • so it seems to me that, in addition to applications querying for available devices, you also want virtual devices to be able to query for apps.

    also, with regards to wanting to use TCP for the virtual device registration protocol: i'm requiring the protocol to be stateful, mostly so that if a client crashes or exits unexpectedly then any devices it had registered get cleaned up properly. TCP gives us some statefulness and there are a few different keepalive techniques i could leverage. it's of course not the only option (indeed, there are several other options listed in the serialosc trello).

  • ideally i'd like both. but i've been thinking that having virtual devices able to query for apps may be more useful.

    if virtual devices, and the manager app they are inside of, could query for apps then it would make working with a managing app much simpler for the user. if in a hypothetical manager app you had made a few pages for different setups, each page having a few virtual devices, i think it would be better to be able to stay looking at the manager app and connect the apps you want to the virtual devices you have created there. rather than having to go to each app and connect it to a virtual device.

    also, as galapagoose's example in max showed, if virtual devices/manager apps can query for apps then you don't need apps to be able to query for virtual devices. the manager app can tell an app to connect to a virtual device, by sending the app the virtual devices receive port, without the app needing to know what virtual devices are available.

  • @karaokaze
    just tested out the ports problem with max for live and straight max. seems they can't both receive on the same port, but can both send to the same port.

    one near solution would be, when a monome app is opened it sends a "/newapp (prefix) (port)" message to port 12003. the manager app receives these /newapp message on port 12003 and stores them, then sends any /ping messages to the monome app ports it has stored. rather than sending /ping messages to 12003, which only one program can listen to at a time it seems.

    this would mean you'd always have to open your manager app before opening any monome apps though... hmm.

  • My mind is blown by this thread. I can't wait to see what you guys come up with.
    (If anything haha)

  • fundamentally what i've tried to demonstrate with this example is that you can create a child-of-serialosc inside of another environment (eg. max). to serialosc proper this Manager app looks like an application, and to the Client app it looks like a device (albeit through an alternate connection method). i think that means it's functioning as a virtual device.

    what this means is that if you build the Manager app like serialosc.maxpat it will clean up after itself solving many of the above problems. ie: dynamic port registration w/ conflict-checking. the serialosc.maxpat already has this coded, it just needs to be translated across with the different port messages.


    also at quite a high level the idea is the Manager app is merely an intermediary between device and app. in this way you could replace the 'serialosc.maxpat' in the Manager with a connection to an alternate grid device (launchpad, apc etc) — indeed i've already created that translation layer inside of mlrv.

    through this method it's quite easy to allow a number of Devices in the Manager app which creates a composite-virtual-grid (non-regular / rectangular grids would be more challenging).

    the necessity for logging currently open apps is born of the fact that i can't think of a simple way to add virtual devices to the main serialosc.maxpat dropdown list in a non-max-specific manner. i would far prefer to have an app that simply spans / routes / pages, which can then be selected inside the Client apps (preserving the language of 'select-from-dropdown' in serialosc.maxpat).

    due to the direct p2p approach of this Manager app we've discussed i feel the most logical approach is to keep it really simple. it makes more sense to me to have an app that splits a 128 into two 64s and maps a separate app to each, and another for attaching two 64s to one application creating a composite-128. this simplifies things greatly in terms of setup and instead provides a number of preset options.

    the idea of an all-in-one paging router/spanner is great but also necessarily quite complex and a more long-term goal. i know phortran isn't interested, but perhaps a budding forum member would be game enough to retrofit this method to it. my guess is it would be a very complicated and hacky solution though...

    finally, this entire concept is specifically based around not-needing serialosc proper to be altered. if there are going to be changes made to serialosc itself, i would strongly encourage the virtual device registration to be handled by the driver — this is entirely a hack and breaks down the language of using serialosc.maxpat to connect to a device directly.

    as such, i think discussion which requires changes to serialosc should be had in a new thread, and this one should be directed solely at creating a virtual-device ('Manager') app that works with the tools we've already got.

  • thinking further on this topic — perhaps it would be possible to make an app that spawned VirtualDevices in the serialosc.maxpat (or other selection method)? would basically use the same method as above with a parallel udp-receive set to 12003 (the Manager apps output).

    would just monitor for /serialosc/add (etc) messages and trigger a refresh of the VirtualDevices list which would be received and added to the device list with a prefix indicating it was a virtual device....

  • not sure if this is the proper thread for this question but it could be

    i've made several max patches that use the prefix /monome
    i plug in 2 physical monomes and connect each to a different app using the prefix /monome
    both applications work properly on each physical device, even though they share the prefix /monome

    any there any issues with sharing /monome across two physical devices (separate applications) simultaneously? good practice? bad practice? perhaps /monome1 and /monome2 is a better practice?

  • it is quite ironic that i'd been advocating for the deprecation of the /prefix command in serialosc. once the transition to serialosc had been made devices were no longer referred to by prefix at all and the connection of device to app is now handled under the hood (rather than monomeserial where you had to enter these details by hand).

    in experimenting with these multi-app browser techniques the usefulness of the /prefix model has reared it's head once again if for no other reason than visually being able to tell commands apart.

    at this stage i'd recommend a short but descriptive /prefix in any new apps you're developing just in case. if your app uses both grid and arc, perhaps we should define a best practice for that (eg. /grid-flin /arc-flin, or /flin-g /flin-a etc.)

    then again this is quite off-topic for this thread so unlikely many other folks will read it...