Clusters of Apple silicon devices- is it possible?

[iPadified]

I am probably not alone with have too many apple devices...It struck me the other day that some some specialised workloads, it would be great to easy create a compute cluster of Apple devices. If they are connected via thunderbolt/USB-C it maybe bandwidth enough to let the Mac use the compute power of the iPad and iPhone connected to it. Similarly, an array of Mac minis would be great.

Any type of work hah can be parallelised should benefit from this. I think of ray tracing of 3D models (they are already now using render clusters), but perhaps also split a video on several parts and let the different devices render one part each.

It would be great to just select which render devices to use directly from FCP. Hence, no difficult handshaking procedures.

What are your thoughts?
Would it be possible?
Would it be useful?

 

[The Mercurian]

Came here to ask similar question 😅

What I was wondering is could you build a cluster of the new M1 Mac Minis to process R code in parallel across machines. On the R side its easy enough to write code to take advantage of a cluster I believe, but I've ont idea what needs to be done to set up a cluster on the network/OS level. @leman have you any knowledge on topics like this ?

 

[pshufd]

We used VAXClusters and VMSClusters in the 1980s and 1990s. The idea is you tie a bunch of computers and storage nodes together with high-speed interconnects and you share system services like batch queues, user credentials, email, etc. You could log into any cluster node and use it to get at cluster-wide services. I'm sure that you could do that with a network today. But you'd have to be able to package up the work in chunks that could get executed across the cluster.

 

[leman]

Came here to ask similar question 😅

What I was wondering is could you build a cluster of the new M1 Mac Minis to process R code in parallel across machines. On the R side its easy enough to write code to take advantage of a cluster I believe, but I've ont idea what needs to be done to set up a cluster on the network/OS level. @leman have you any knowledge on topics like this ?

There is a bunch of packages that allow you to set up distributed computing for R, I think the core parallel already has functionality like this. I never really had any use for it myself. For practical purposes, I think splitting the work into batches and then using some sort of distributed work scheduling system makes more sense. R itself is not the most robust software in the world.

 

[The Mercurian]

There is a bunch of packages that allow you to set up distributed computing for R, I think the core parallel already has functionality like this. I never really had any use for it myself. For practical purposes, I think splitting the work into batches and then using some sort of distributed work scheduling system makes more sense. R itself is not the most robust software in the world.

Yeah I've used parallel and future packages alot for single machine parallelisation - to reconfigure that side of it for cluster is super easy just change an argument, but it needs backend support too. R Studio Server can do it I am told but thats linux based.

I think the bottom line here is the MacOS won't support cluster computing.

 

[dreikelvin]

Came here to find this thread 2 years later. With the new iPhone 14 Pro, I would love being able to make the most out of its computing power which otherwise would never be utilized just in a standalone device. Could hooking this up via lightning to a mac studio or mac mini M1 turn it into a mini cluster for GPU/CPU intensive tasks like generating StableDiffusion images or rendering in Blender? This would be awesome...

 

[Xiao_Xi]

You can train a deep learning model using multiple Mac Studios. Apple gave a demo at the last WWDC. The demo starts around 14:30

Accelerate machine learning with Metal - WWDC22 - Videos - Apple Developer

Discover how you can use Metal to accelerate your PyTorch model training on macOS. We'll take you through updates to TensorFlow training...

developer.apple.com

 

[dreikelvin]

I am not a programmer but does this mean the software capabilities are there and all we need is some developer implementing this into let's say a plugin in blender or steinberg can implement this into cubase, detecting a connected idevice/mac and then make the app extend its workflow to the connected devices?

 

[altaic]

Signs point to yes, but tip your developer.

 

[leman]

I am not a programmer but does this mean the software capabilities are there and all we need is some developer implementing this into let's say a plugin in blender or steinberg can implement this into cubase, detecting a connected idevice/mac and then make the app extend its workflow to the connected devices?

“Software capabilities” just means that computers are connected to a shared network or can otherwise communicate. One certainly can implement functionality like you describe, it’s just not always going to be useful or worthwhile.

 

[Boomhowler]

It shouldn't be impossible to setup Kubernetes on the arm Macs and then submit jobs to that cluster. Getting it to run locally is super-easy, just install Docker Desktop, but then you have to make sure that the R-code that you want to run support Kubernetes (or k8s, as it is called) ways-of-working.

 

[dreikelvin]

Wished it was a little more straight forward. Like, plug your iphone into your mac and pull up system-preferences-straight-forward

I will probably never use my mac for compiling code or anything Docker-based. Just rendering my audio projects or doing a quick Blender render/Stable Diffusion render.

Ah well

 

[Xiao_Xi]

It shouldn't be impossible to setup Kubernetes on the arm Macs and then submit jobs to that cluster. Getting it to run locally is super-easy, just install Docker Desktop, but then you have to make sure that the R-code that you want to run support Kubernetes (or k8s, as it is called) ways-of-working.

The best way should be to run Kubernetes on metal like MacStadium does with Orka.

ORKA: Orchestration with Kubernetes on Apple

Orka Platform enables macOS orchestration and life cycle management in a cloud environment by leveraging Kubernetes to deliver high-performance virtual macOS environments in seconds for everything from simple Xcode builds to fully integrated, complex automated CI/CD pipelines.

www.macstadium.com

However, if you want to run Kubernetes on macOS, you can also use Rancher Desktop.

Rancher Desktop by SUSE

Open source desktop application that provides Kubernetes, Container Management, bundled utilities on the desktop

rancherdesktop.io

Depending on your workflow, you can use Spark to distribute your R jobs.

Running Spark on Kubernetes - Spark 3.5.3 Documentation

sparklyr

spark.rstudio.com