Midi video on my Pi

1

I’m sure this has been discussed but a personal response from the community would do wonders for me actually tackling the project.

My projector setup has live cameras, VCRs, DVDs, HDMI, all mixing together in harmony. Now it’s time for me to have a midi-controllable video clip launching and manipulation app.

I know about recur and will use it if it is my best choice.

But Ideally:
Banks of 16 clips viewable as thumbnails
Play retrigger scrub rev ff etc
Process/capture realtime

Yes that’s like the Edirol p-10’s functionality and if that is the best way for me to do it then I’ll save up. It’s expensive though.

How about this liVES program? It looks amazing on paper and I feel like I don’t hear people discuss it much. I have tried installing it on various versions of Raspbian and Ubuntu on my 3B+ and 4B but get 404 errors from servers and other problems.

But I’m not here to troubleshoot my installation. Does anyone use this in 2022 on a Pi? Should a novice such as myself be able to figure out?

I’m keeping my setup sans-computer.

Any wisdom appreciated

2

from my experience implementing the video ‘scrubbing’ / ff / backwards is quite tricky - it depends what kind of video files you want to playback, but most are compressed and containerised (w keyframes) in a way that can only really be played forward…

one way I was able to get around this is to load every frame into memory, and then you can flip through this array of frames however way you like (kind of like flicking through doodles on corner of a notebook) - the limitation of this approach is the length - 60 seconds or so is possible depending on the pi, resolution & other factors …

3

keep in mind also the limitations of the p10 re resolution and the non-trivial process for converting files. Perhaps a diy rpi solution that was restricted to using especially converted files for it would have more luck on the scrubbing // reversed features ?

anyway best of luck ! I have not heard of liVes but interested in it now.

4

Agreed. Files would need to be encoded with a keyframe every frame, and for the case of codecs like MP4, no B-frames. Encoded this way, they may not have to be loaded into memory if the file system is fast enough to read them. So, some slower USB drives/SD cards might not be able to handle the throughput.

5

Interesting. So based on these responses there is not a software that does this and it would take intensive keyframing for every frame. Not what I expected to hear.

I bought a motion dive.tokyo in an attempt to accomplish it. I never got it to work even though I went so far as to buy a vintage laptop and find the trial software. But that system seriously old and scrubs like a champ.

About the 60 second theoretical limit - that’s great it’s a lot more than I need. Just short clips to be played back on demand is what I want.

I’m pretty sure what I read about liVES said it can do that and more. I would be surprised to find out there is an intensely laborious prep stage to get it done. But I guess I’m surprised about everything at this point.

The p-10 - I am quite happy with the low resolution. Everything is SD in my system. When considering buying it I figured I would capture and trim the clips with the machine itself. Not the best not the worst.

About card read speeds - I figured the data would be in RAM no matter what and at SD resolution the Pi should have more than enough to handle it.

So on the Pi there is not an established program to do this? I’m surprised. But I’m here for help and if you say so that’s my working knowledge for the time.

6

Motion-JPEG is probably the codec to use as it is really just a bunch of JPEG’s strung together. Pretty easy to decompress/play and every frame is a complete, standalone entity (no B-frames here). It is very easy to encode/decode so CPU overhead should be minimal. Avoid files that require lots of CPU to decode (MP4, h.264, etc)

Hmmm, I should have a go at it and see what I can come up with…

7

That is very cool.

If I can muse for a minute:

It seems to me that the best basic set of functions for a “midi video clip launcher” would be:

Launch/retrigger - forward/reverse speed - scale - height - width - x position - y position - rotation - flip vertical - flip horizontal - toggle loop on/off - start pos end pos - panic/clear clip attributes

For simplicity, one channel, no internal mixing. Static thumbnails on monitor would suffice. SD for feasibility.

CC#s 1-16 set to toggle (>63). They launch/retrigger clips 1-16. Last clip launched is now able to be manipulated with midi cc.

CC#17 - speed
-400% to -1% (0-63) 0% (64) 1% to 400 (65-127) *this is effectively forward/reverse, and 0 speed is effectively pause.*

CC#18 - scale controls height and width simultaneously
(0-127) 0 is a single pixel and 127 is blown up to many times its size.

CC#19 - height
(0-127)
CC#20 - width
(0-127) These two are the same idea as 18 but individually controllable. Can cause distortion

CC#20 - x position
(0-127)
CC#21 - y position
(0-127) these two are self-explanatory

CC#22 - loop/one-shot toggle
(0-63,64-127) self-explanatory

CC#23 - start position
(0-127) 0 is the normal start position. 127 is the last frame.

CC#24 - end position
(0-127) same concept. 23 and 24 could potentially interact poorly but it doesn’t matter, the operator should avoid doing that.

CC#25 - flip horizontal
(0-63 normal, 64-127 upside down)

CC#26 - flip horizontal
(0-63 normal, 64-127 mirror)

CC#27 - rotation
(0-127) rotate it

CC#25 - Panic/clear clip attributes
(>63) reverts clip back to imported condition.

CC#26 - bank select

PC#s - preset ie a specific clip with multiple attributes applied

And the last core function would be tiling. That strikes me as being way too complex to implement! Not that any of this sounds remotely easy. But that feature seems pipe-dreamy. But an lfo automating cc#20 (x position) would be 100x cooler with tiling.

A host of other options from brightness to keying would be sweet of course. But this would be a single, reliable, stripped down piece of gear that I would use every day.

I’m not posting this with expectations of it being made of course. I enjoy the mental exercise of thinking about what a “lean” yet effective device could look like.

8

On CC#17 I typed:

This is effectively forward and reverse, and 0% is effectively pause.

9

Maybe go for Resolume? HDMI > SD converter into the chain? You can run earlier versions of resolume on lower-power computers, and maybe if you go for a smallish gaming laptop … I totally get wanting to stick with Pi but I think it’s kind of hard to beat a little laptop even if just for versatility’s sake

10

Resolume eh?

It looks extremely functional, as a lot of computer stuff is. But it opens several cans of worms.

Functionally speaking, if I add Resolume, I may as well use its effects. At this point I’ve already superseded much of the work I’ve put into my setup. And if I’ve got the computer in the mix, I may as well get all the functionality that I can out of it. And why not a second laptop? I have several ready for the task. A whole new world :musical_note:

Philosophically speaking, I’ve come this far sans-computer…

But why do I care about that? If what I want is the “best” product why don’t I pre-process the footage in After Effects? Why do any of this when I could prefab videos and project them straight from a thumbdrive? In my case I have live cameras in the mix, so my setup is at least somewhat justified as being pragmatic? :sweat_smile:

There is a tangible benefit of the simplicity and reliability of going sans-computer. My Waaave_Pool boots up and acts like a piece of hardware. All of my video gear is on one power cord. When I plug it in fans whir and lights blink, power buttons already engaged. When I’m done I harshly pull the plug out of the wall. It almost acts like a single unit.

And importantly — suffice it to say that computers behaving badly have made a fool of me more than once. I :heart: hardware.

I have an elaborate midi music setup and wrestle with the same issues with it.

If an Eyesy-like box could be my clip launcher I would be thrilled. Now that I’ve talked about it, launching, retriggering, and one shot/looping from a bank of 16 clips would be a quantum leap for my setup and keep me happy for a while. Even if I’m not reversing, speeding up, or anything remotely fancy.

11

…maybe have a look at GitHub - cyberboy666/r_e_c_u_r: an open diy py/pi based video sampler from ‘our’ @cyberboy666 ?..

12

Well yeah recur looks great! As I mentioned somewhere in my wall of text above if that’s the best choice I will just go for it. I would love to have something up and running.

From my limited knowledge of recur -

It doesn’t seem to have thumbnails. Is that right? I could deal with that if I have to.

The bank is less than 16?

I’ve seen posts on this forum saying midi is difficult to tailor? I don’t plan on using the keypad. The 16 pads on my beatstep pro with all their sequencing glory are what I want to use right from the rip.

13

Right on it did not occur to me that I’m talking to the author of recur. I’m hugely impressed with your work!

I’ve been looking at that github for years but have suffered from analysis paralysis until like 2 days ago.

1 Like
14

if you already have a rpi and a midi controller its not a huge investment to give it a go and see how it works for you - you will just need to pick up a 3.5" tft gpio screen…

  • it doesnt have thumbnails, but depending on your setup there is a way to preview clips, using screen mirroring , also it handles nested folder structures so (imo) a well organized clips library does the trick :wink:
  • there are infinite banks of clips possible - each with 10 sample slots. in theory you can flatten these banks onto ur controller mapping as discussed here
  • in my experience the midi is very easy to tailor (but i am biased since i made it that way lol), one option is to just use the default midi mapping table and edit what your controller outputs using their software to suit your needs.
  • or if you prefer to edit mapping inside recur there is a mapping json file you can edit to suit your needs (its even possible to create a new mapping per controller and have this automatically loaded when that type of controller is plugged in - although maybe thats getting ahead of ourselves haha)

and finally, since the source is open, you are more than welcome to make the changes you want to create your perfect tool ! or take inspiration from what is here to build something new that meets your needs. im happy to see musings over how you would design your perfect tool - its the first steps to taking that power into ur own hands :fist: :hammer_and_wrench: :tv:

2 Likes
15

Wow. That is some good info right there.

As soon as I have a spare minute I will get that midi mapped and finally try it! Big moment.

I’m interested in everything you wrote there. I will use it to live my best video life, thank you.

The responses and encouragement I have gotten so far are invaluable, I feel on track.

16

i would reiterate that recur is yr best choice for video sampler at the moment. resolume is also incredibly handy and quite peformable (and easily piratable) and worth taking a peek at, one could simply pick up a cheaper laptop (if yr processing in SD you can get away with a lot for a little), set up a bunch of midi macro controls for stuffs, and invest in a decent quality downscaler to get analog video outputs. running touchdesigner on a little windows box could be a good option as well.

i have done various kinds of work implementing video samplers on pi3bs for commissions and personal projects, most everything that folks would want is theoretically possible but usually at a performance cost that wouldn’t really work with most folks worflows. assuming that the VSEJET (platform for gravity_waaaves) works out as a general development environment for my future projects, i would be interested in putting together some kind of live video sampler/looper/resampler. My goals would be more along the lines of something that functions more like a rc-505 but with resampling and more of an elektron style approach to processing video. I don’t personally work with clips enough in performance to be motivated enough to implement anything like a p10 without commissions, but i do enjoy recursively layering and processing clips with weird ass open cv/optical flow style stuffs enough to focus time and energy on developing that angle of things…

2 Likes
17

Awesome, thank you.

I’m going to try to keep myself computer-free but am well-aware that bending over backwards and ultimately crippling myself isn’t the best way to live my life.

But I’d better start by actually trying out what is available to me. So all of this has given me the gumption to get recur up and running. Knowing is half the battle! I’m on it.

Your response gives me perspective. Launching many short clips with midi as part of a live musical performance is perhaps more niche than I thought. The bands I know who have projection usually line up a long video and play to a click track.

The SD-ness of this is the only reason I can get so much done with low-powered gear and I love that. It plays so well with ccd cameras, which add a an imperfect sheen that just couldn’t be more perfect. I told myself in this endeavor that even if it never sees fruition, I’m going to do this and enjoy the aesthetics and experience of the process and it’s very good for me mentally.

Staring paralyzed at githubs with a Pi in my hand has been a thing. Now staring at short clips will be my Wednesday.

Thanks again

18

…you might want to check out what @YOVOZOL does - he also has a video online where he explains how he uses Resolume (to launch clips) together with his ‘analog gear’…

…ref. https://www.youtube.com/@YOVOZOL - and perhaps it was this video (i am not sure):

19

Seems like a good resource! I’ll keep wrestling with the computer question.

I’ve come this far and am in love with my system, I just want midi clips. (I just call them midi clips now :man_shrugging:) The Waave_Pool is in the guts of this and I “never think about it.” I would love more items like that.

image
image1920×1440 413 KB

20

And I guess I’d like to add that I edit and mix audio on my computer many hours every week and seeing the timelines on Resolume makes me say, “yay more squinting and clicking.”