I have also a spark cluster created with google cloud dataproc.
Source:
over 1 year ago
Specifically, we heavily rely on managed services from our cloud provider, Google Cloud Platform (GCP), for hosting our data in managed databases like BigTable and Spanner. For data transformations, we initially heavily relied on DataProc – a managed service from Google to manage a Spark cluster.
– Source: dev.to
/
over 2 years ago
With that, the best way to maximize processing and minimize time is to use Dataflow or Dataproc depending on your needs. These systems are highly parallel and clustered, which allows for much larger processing pipelines that execute quickly.
Source:
over 2 years ago