Show HN: SQLFrame – I ran PySpark without Spark on a SQL database Recently I open-sourced SQLFrame, a DataFrame library that implements the PySpark DataFrame API but removes Spark as a dependency. It does this by generating the corresponding SQL for the DataFrame operations using SQLGlot. Since the output is SQL this also means that the PySpark DataFrame API can now be used directly against other databases without the Spark middleman. I built this because of two common problems I have faced in my career: 1. I prefer to write complex pipelines in PySpark but they can be hard to read for SQL-proficient co-workers. Therefore I find myself in a tradeoff between maintainability and accessibility. 2. I really enjoy using the PySpark DataFrame API but not every project requires Spark and therefore I'm not able to use the DataFrame library I am most proficient in. The library currently focuses on transformation pipelines (reading from and writing to tables) and data analysis as key use cases. It does offer some ability to read from files directly but they must be small although this can be improved over time if there is demand for it. SQLFrame currently supports BigQuery, DuckDB, and Postgres with Clickhouse, Redshift, Snowflake, Spark, and Trino in development or planned. You can use the "Standalone" session to test running against any engine supported by SQLGlot but there could be issues with more advanced functions that will be resolved once officially supported by SQLFrame. Blog post for more details: https://ift.tt/U0pTQN4... Would love to answer any questions or hear any feedback you may have! https://ift.tt/gEPLHpo May 21, 2024 at 06:39AM
Show HN: SQLFrame – I ran PySpark without Spark on a SQL database https://ift.tt/4liObUq
Related Articles
Show HN: Track and trace cargo across major shipping lines https://ift.tt/37VeuvhShow HN: Track and trace cargo across major shipping lines https://ift… Read More
Show HN: Write For Your Reader, a weekend project for nested writing https://ift.tt/3rSbMifShow HN: Write For Your Reader, a weekend project for nested writing h… Read More
Show HN: DFlex – JavaScript Project to Manipulate DOM Elements https://ift.tt/2LcZPCLShow HN: DFlex – JavaScript Project to Manipulate DOM Elements https:/… Read More
Show HN: Cyberpunk Trainer – My First OSS Project https://ift.tt/3aFGlBqShow HN: Cyberpunk Trainer – My First OSS Project Hey gang, My names R… Read More
Show HN: MIDI to Blob Opera Converter https://ift.tt/34C1msUShow HN: MIDI to Blob Opera Converter https://ift.tt/37IInyH December … Read More
Show HN: Diffgram – Human Supervision of AI Data https://ift.tt/3o4iRtzShow HN: Diffgram – Human Supervision of AI Data https://diffgram.com/… Read More
Show HN: Duuce – marketplace for buying/selling newsletters https://ift.tt/2MrbzSJShow HN: Duuce – marketplace for buying/selling newsletters https://du… Read More
Show HN: Airbnb for Musicians https://ift.tt/3rPzZFGShow HN: Airbnb for Musicians https://noisycamp.com January 3, 2021 at… Read More
0 Comments: