Learning Spark

This book is a great resource to learn about spark. It covers in detail the concepts related to the Spark architecture, theoretical concepts about parallelization and topics related to optimizing analytical pipelines running on Spark. The book has a very nice section about the delta lake. Also covers MLflow yup a good level of detail, more like a complement to the docs. The section on machines learning includes theoretical explanations on how some ML algorithms change when running then parallely, as MLlib does. I used the book as an extra study resource when taking some Databricks certifications. It was a great addition to my study materials.

Da buenos ejemplos sea en Scala y python aunque no siempre están en python el lenguaje Scala es similar (como un Java python). Sugiere que si quieres practicar utiliza databricks si no quieres instalar nada on-premise o si gusta instala spark utilizando wsl de Windows o una máquina virtual con Linux.

This book is a must read for anyone trying to learn Spark in the big data environment.

I am always trying to learn new skills to make myself more marketable in the work place. My background is mainly in SQL with some Python and I am learning JS right now. I decided to give this book a shot to see whether Spark is another tool I want to add to my arsenal. The books does what it promises; it gives you a good introduction to Spark. I did have some issues installing the required programs on a MacBook, but once I had everything installed, I was able to follow along. My big complaint is what others have mentioned, which is concepts are mentioned without any background to what or why. If you have some programming background, this book should be sufficient to get you up and running in Spark.

Nice book if you really want to work hands on without having to worry about internals of spark.

I'm a software engineer who knows his way through SQL, mostly running queries/transforms on Postgres and Redshift. The majority of my background is in building and supporting services. Having no background knowledge in Spark, I was looking for a book that explains the fundamental concepts, helps me get up running, and helps me expand my toolkit for working with "big data". I was able to follow along in this book fairly easily. Working on a MacBook, I did have to first install Scala, download Spark, enable Spark in IntelliJ, etc. I didn't have trouble with this as it was fairly straightforward. With my environment set up, I found the book presents every code sample in Scala and Python. I worked through the code samples, chapter by chapter, writing Scala in IntelliJ or sometimes writing Scala in the Spark CLI itself. I did take a detour from the book slightly to learn a bit more about sbt, which is the Scala build tool. For a beginner such as myself, this book is a God send, but I do wish the authors approached some things differently. In my opinion, some topics are covered in a very "hand-wavy" manner. For example, Chapter 4 discusses managed vs. unmanaged tables. While knowing this difference exists is helpful for the reader, the authors never discuss when you should use a managed table or an unmanaged table. They could have included that information or pointed the user to some external source. This part of Chapter 4 then shows sample code on how to create a managed table from a CSV file. However, it's not clear what should I do with that information. What are the patterns applicable to a managed table vs. unmanaged table? What are the trade-offs? Being a beginner book, I still feel the authors could have written even just 1 page, which would add significant value to this section. Sometimes the book will share some interesting tidbit but using terminology or concepts that the authors haven't really described. I found this very frustrating. For example: > (Chapter 4, page 92) ... you can create multiple SparkSessions within a single Spark application—this can be handy, for example, in cases where you want to access (and combine) data from two different SparkSessions that don’t share the same Hive metastore configurations. If you search for mentions Hive, you see the authors briefly mentioned Spark uses a Hive metastore to persist table metadata. So are the authors saying I can use one Spark installation and access table metadata from different Hive metastores? Why would I ever want to access only the metadata for different tables? Again – the use case isn't clear. As a beginner, I found this book very valuable, and I believe it is a great investment.

I am a beginner in Spark, so this book helped me to get quick start in Spark. Ch 1-2 - good intro and quick steps to run your first spark word count code sample. Ch 3-6 - Spark APIs, Spark SQL, Data Frames Ch 7 - good tips in tuning and optimzing Spark Apps, e.g. view/check configs, UI, static vs dynamic resources allocation, config Spark executors’ memory and the shuffle service, Caching and Persistence of Data, Jobs and Stages , Debugging Spark applications. Ch 8 - Structured Spark Streaming, Streaming Query, Streaming Data Sources and Sinks, Stateless and Stateful Transformations Ch 9 - Building Reliable Data Lakes with Apache Spark: storage solutions, databases and data lakes, lakehouses which is the new storage solution with Transaction support, Support for upserts and deletes, Schema enforcement and governance, etc. Ch 10 - Machine Learning with MLlib: ML models using MLlib (the de facto machine learning library in Apache Spark), best practices for distributed ML and feature engineering at scale Ch 11 - Managing, Deploying, and Scaling Machine Learning Pipelines with Apache Spark: MLflow to track, reproduce, and deploy your MLlib models, best practices for model management to get your models ready for deployment. MLflow is important in model management ! Ch 12 - Apache Spark 3.0 : It is now officially released with Spark 3.0.1 released (Sep 08, 2020). You can see new 3.0 features in this chapter, e.g. adaptive query execution; dynamic partition pruning, etc...

The book is very easy to follow and the first five chapters get you up and running. I recommend getting the databricks community edition to type in the examples. That way you can focus on learning Spark instead of trying to work with all the infrastructure considerations. Chapter 6 is very relevant, if you are a scala dev you can get into the weeds there. There's even a chapter dedicated to Delta lake which I think is a must read. The MLlib/ Mlflow chapters are pretty much for the novice, so it's a good segue for the data engineer to get a gentle intro to ML. Overall I highly recommend the book.

Highly recommend this book for beginners looking to get into Spark programming. Examples are shown with both Python and Scala. I found the authors writing style extremely pleasant. Being a technical book the explanations were very easy to follow. Complicated technical terms are explained in very simple english. I have the kindle edition and noticed that the formulas on one of the pages on machine learning was slightly cutoff at the edges but I wont remove a star because of that. In my view there are tons of material online to understand those regression formulas. What really worked for me is how great a job the authors have done in explaining how to use Spark 3.0. Since I am a Python and SQL user this book really benefits me at work. The syntax and function explains are very clear and with an online Databricks account one can really practice as you learn with an uncomplicated dataset. How to program the Dataframe API is really well covered.

You should probably have some familiarity with machine learning and python before you pick this book up, but it's a decent introduction to Spark.

Bardzo solidna pozycja

I've just started my role as a Data Engineer where I looked at Azure's Data Factory. I needed to learn PySpark so I picked up this book and found it a super useful guide. It is explained clearly, and whilst it's clearly aimed at someone who has been in the industry longer than I, I found I could easily understand it. I haven't read the chapter on streaming or the two chapters on machine learning as it isn't applicable to me, but everything else has been just what I needed. Well done to the authors for putting together such an amazing guide. If you want to see the different chapter contents, I've added them as photos for your ease.

Buen libro como complemento de “Spark The Definitive Guide”, que para no es la referencia principal. Este libro da una versión resumida del que acabo de comentar con unos cuantos ejercicios adicionales que pueden ser bastante útiles para la comprensión de la teoría.